Whole-school interventions promoting student commitment to school to prevent substance use and violence, and improve educational attainment: a systematic review

Search terms

A draft search strategy was compiled in the OvidSP MEDLINE database by an information specialist (JF). This included strings of terms, synonyms and controlled vocabulary terms (when available) to reflect:

1. population (children and young people aged 5–18 years attending school)

2. intervention (whole-school interventions aiming to prevent violence or substance abuse)

3. evaluation methods.

These concepts were combined using the Boolean operator AND.

Search terms were determined via a text analysis of 77 known articles featuring terminology relevant to the review,58–132 using NVivo 12 Plus software (QSR International, Warrington, UK). Titles, abstracts and keywords were imported and word frequency and cluster analyses were used to create an OvidSP MEDLINE search string that included common subject headings and incorporated relevant words in proximity. These were augmented by additional terms and synonyms so that the search terms adequately described the search topics. These were then combined together and tested systematically. 133 This search strategy was refined with the project team until the results retrieved reflected the scope of the project. When run in an education-focused database, rather than a health one, it became apparent that the sensitivity-to-specificity ratio of the search was too high, resulting in many irrelevant results. Therefore, the search strategy for concept 2 was narrowed. This search was peer reviewed by a librarian not involved with the project using the Peer Review of Electronic Search Strategies guidance. 134 The agreed OvidSP MEDLINE search was adapted for each database to incorporate database-specific syntax, controlled vocabularies and search-interface limitations. Details of the search strings used for each database can be found in Report Supplementary Material 1, and in the London School of Hygiene & Tropical Medicine’s data repository. 135

Databases

The following databases were searched in full between 16 and 27 January 2020:

• ProQuest Applied Social Sciences Index and Abstracts (ASSIA) (1987 to 23 January 2020)

• ProQuest Australian Education Index (complete database as of 23 January 2020)

• EBSCOhost British Education Index (complete database as of 16 January 2020)

• EBSCOhost Cumulative Index to Nursing and Allied Health Literature (CINAHL) Plus (complete database as of 16 January 2020)

• Wiley Online Library Cochrane Library (issue 1 of 12, January 2020) was used to search for results from the following databases:

  • Cochrane Database of Systematic Reviews

  • Cochrane Central Register of Controlled Trials

• Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) database of health promotion research (BiblioMap) (complete database as of 23 January 2020)

• EPPI-Centre Database of Promoting Health Effectiveness Reviews (DoPHER) (complete database as of 23 January 2020)

• OvidSP EconLit (1886 to 9 January 2020)

• EBSCOhost Education Abstracts (HW Wilson) (complete database as of 17 January 2020)

• ProQuest Education Database (complete database as of 24 January 2020)

• EBSCOhost Educational Administration Abstracts (complete database as of 17 January 2020)

• EBSCOhost Education Resources Information Center (ERIC) (complete database as of 17 January 2020)

• OvidSP EMBASE® (1947 to 14 January 2020)

• OvidSP Global Health (1910 to 2020, week 1)

• OvidSP MEDLINE ALL (1946 to 14 January 2020)

• OvidSP PsycInfo (1806 to 2020, week 1)

• Elsevier Scopus® (complete database as of 16 January 2020)

• OvidSP Social Policy & Practice (October 2019)

• Clarivate™ Web of Science™ and Social Sciences Citation Index (1970 to present; data last updated on 9 January 2020)

• EBSCOhost Teacher Reference Center (complete database as of 24 January 2020).

These databases were selected to retrieve literature from the fields of health and education. We amended the list of databases originally intended to be searched (see Appendix 1, Table 8, for deviations from, and clarifications of, protocol) on the advice, informed by initial pilot searches, of the information scientist (JF).

The searches were updated between 11 and 19 May 2021. The search strings used for the updated searches can be found in Report Supplementary Material 1. The updated searches used the following databases:

• EBSCOhost CINAHL Plus (complete database as of 11 May 2021)

• Wiley Online Library Cochrane Library (issue 4 of 12, April 2021) was used to search for results from the following databases:

  • Cochrane Database of Systematic Reviews

  • Cochrane Central Register of Controlled Trials

• EPPI-Centre database of health promotion research (BiblioMap) (complete database as of 13 May 2021)

• EPPI-Centre DoPHER (complete database as of 13 May 2021)

• OvidSP EconLit (1886 to 29 April 2021)

• EBSCO ERIC (complete database as of 19 May 2021)

• OvidSP Embase (1947 to 10 May 2021)

• OvidSP Global Health (1910 to 2021, week 18)

• OvidSP MEDLINE ALL (1946 to 10 May 2021)

• OvidSP PsycInfo (1806 to May 2021, week 1)

• Elsevier Scopus (complete database as of 13 May 2021)

• OvidSP Social Policy & Practice (January 2021)

• Clarivate Web of Science and Social Sciences Citation Index (1970 to present; data last updated 13 May 2021).

Owing to COVID-19 restrictions, visitor access to libraries was not allowed. Therefore, it was not possible to update searches on the following databases:

• ProQuest ASSIA

• ProQuest Australian Education Index

• EBSCOhost British Education Index

• EBSCOhost Education Abstracts (HW Wilson)

• ProQuest Education Database

• EBSCOhost Educational Administration Abstracts

• EBSCOhost ERIC

• EBSCOhost Teacher Reference Center.

Outcomes targeted by interventions

Of the 22 interventions included in the overall review, 8 targeted violence outcomes only,69,73,161,169–171,178,191,192,197,199,200 and 1 focused on substance use only. 175,176 Seven targeted both violence and substance use. 66–68,103,115,117,165–168,180–188,190,194–196,200 Two interventions targeted violence and academic outcomes,80,89,98,172,189 and two targeted substance use and attainment. 174,193 Two interventions targeted violence, substance use and academic outcomes. 52,61,63,64,85,108–110,113,123,124,162–164,173,179,198

Phase of schooling targeted by interventions

Of the 22 interventions covered by the reports included in the review, 5 (23%) targeted students broadly corresponding to English primary school age (between 5 and 11 years). 162–164,169,171,178,192,197,198,200 The Friendly Schools,169,197 Friendly Schools, Cool Kids Taking Control197 and Friendly Schools Friendly Families (FSFF)171,192 interventions were delivered in Australian primary schools, which typically accommodate students aged 4–12 years, differing slightly to the English primary school phase. Twelve interventions (54%) targeted students corresponding to the English secondary school phase (i.e. aged 11–18 years)66–69,71,73,88,103,115,117,161,165–168,175,176,180–188,190,191,193–196,199 and five (23%) targeted students across both phases (Figure 5). 52,61,63,64,80,85,87,89,98,108–110,113,123,124,170,172,173,177,179,189

FIGURE 5.

Phase of schooling targeted by interventions.

Providers and delivery of interventions

In all the interventions included in the review, implementation was primarily led by schools themselves, with additional support from external agencies or the research team in the form of training only in the case of 6 interventions,169,171,185–188,191–193,197,199 external facilitation only in the case of 1 intervention178,200 and both training and external facilitation in the case of 15 interventions. 52,61,63,64,66–69,71,73,80,85,87–89,98,103,108–110,113,115,117,123,124,161–168,170,172–177,179–184,189,190,194–196,198

Consistent with an emphasis on whole-school change, 17 interventions evaluated in included studies involved the institution of school ‘project teams’, ‘action groups’ or ‘committees’ to engage members of the school community in making decisions about, and overseeing implementation of, intervention activities. These involved school staff only in five interventions;162–164,169,170,175,176,197,198 a mix of school staff and students in four interventions;71,80,88,89,98,165–168,172,191,193–196,199 school staff, students and parents in a further four interventions;52,61,63,64,69,73,85,108–110,113,123,124,173,178–181,184,200 and school staff, students, parents and members of the wider community in two interventions. 87,174,177 In one further intervention, the project team included only staff and a parent representative,171,192 and in another only staff and community representatives. 66–68,115

Of the 13 interventions involving a curriculum element (see Inclusion of curriculum components), 11 had lessons delivered by trained school staff,52,61,63,64,66–68,71,85,88,108–110,113,115,123,124,162–171,173,175,176,179,182,183,191,192,194–199 whereas in 1 intervention, the Aban Aya Youth Project (AAYP) school/community intervention, lessons were delivered by university health educators. 87,177 A further intervention, the Drug Abuse Resistance Education (DARE) Plus programme, involved curriculum delivery by police officers and peer-delivered curriculum elements. 103,117,190 DARE Plus also included neighbourhood action teams aimed at improving community environments, which was facilitated by trained neighbourhood volunteers. 103,117,190

Six interventions involved some element of peer-to-peer delivery. 80,89,98,103,117,172,174,180,181,184,190,191,193,199 This included students acting as peer mentors in Cyber Friendly Schools (CFS) and Portland Peers Project (PPP);191,193,199 students designing and delivering events and awareness-raising campaigns concerning violence and substance use for their peers in the DARE Plus programme, the Strengthening Evidence base on scHool-based intErventions for pRomoting adolescent health (SEHER) programme, Good School Toolkit (GST) and Project Positive Action Through Holistic Education (PATHE);80,89,98,103,117,172,174,180,181,184,190 and, in the case of DARE Plus, as previously mentioned, students delivering classroom curriculum to their peers. 103,117,190

Inclusion of curriculum components

In addition to including one or more of the whole-school components specified in our inclusion criteria (see Chapter 2, Inclusion criteria for the review), over half (n = 13) of the interventions included in the review also contained health curriculum components as part of multicomponent strategies to address violence and/or substance use. 52,61,63,64,66–68,71,85,87,88,103,108–110,113,115,117,123,124,162–171,173,175–177,179,182,183,190–192,194–199 As well as providing health information, 12 interventions also included curricula with a focus on social and emotional learning. 52,61,63,64,66–68,71,85,87,88,103,108–110,113,115,117,123,124,162–171,173,177,179,182,183,190–192,194–199 In five interventions, curricula also addressed bystander action in relation to bullying and aggression. 169–171,191,192,197,199

Use of student data

Eight interventions involved the use of recently collected student data to highlight local priorities and tailor the implementation of various intervention activities to student needs. 66–69,71,73,88,115,165–169,171,175,176,178,192,194–197,200

Duration of interventions

The shortest intervention duration was 8 months. 178,200 Two interventions (9%) were 1 year in duration, 69,73,189 and one intervention (5%) had a duration of 18 months. 80,89,98,172 Ten interventions (45%) were implemented over 2 years;103,117,169,170,175,176,180,181,184–191,193,197,199 six interventions (27%) were 3 years in duration,66–68,71,88,115,162–168,171,174,182,183,192,194–196,198 with one of these (Learning Together) truncated to 1 year in the pilot study included in the synthesis of process evaluations. 71,88 Only one intervention (5%) had a duration of 4 years. 87,177 A further intervention (Positive Action) had an open-ended duration, but was implemented for at least 2 years in each of the studies included in the review,52,61,63,64,85,108–110,123,124,173,179 apart from in the pilot implemented in south-east England, where the intervention ran for 9 months (Figure 6). 113 A descriptive overview of each of the 22 interventions is provided in Appendix 2, Table 9.

FIGURE 6.

Duration of interventions.

Alignment with the theory of human functioning and school organisation

Despite only one intervention being explicitly informed by the theory of human functioning and school organisation,46,71,88,165–168,194,195 a number of themes that were identified in our analysis as reciprocally translating across theories of change in this group aligned closely with the a priori themes suggested by the theory of human functioning and school organisation. The themes apparent across theories of change also enabled some refinement and disaggregation of the a priori themes developed on the basis of the theory of human functioning and school organisation. There were also examples of refutational synthesis apparent whereby some theories of change involved concepts that diverged from those in other theories of change in this group and/or from those suggested by the theory of human functioning and school organisation. 46 These are described to add differentiation and nuance to the analysis. We present a logic model summarising this refined theory of change in Figure 7.

FIGURE 7.

Logic model for school environment interventions developing student/parent participation and enhancing relationships.

Reframing school organisational management

Elaborating our a priori concepts, the reframing of school organisational management to involve students, parents and other members of the community in school-level decision-making emerged as an overarching theme recurring across descriptions of intervention theories of change. A number of interventions involved surveying students66–69,71,73,88,115,165–169,171,175,176,178,192,194,195,197,200 or consultation with students to identify priorities for action on school improvement,180,181,184 thus making school-level decision-making more inclusive of student views and ideas. This was described by Shinde et al. 181 in their medium-quality account of the theory of change for SEHER:

[T]he speak-out box was a letterbox providing a platform for students to raise concerns, complaints, and suggestions anonymously … issues were addressed through … discussion during various activities of the intervention.

Other strategies for making school decision-making more student-centred involved students participating directly in groups with school staff to make decisions about changes to school learning or discipline systems. 69,71,73,80,87,88,98,165–168,172,174,177,178,180,181,184,191,193–195,199,200 Although such approaches were theorised to make school provision and policies more student-centred and to support the implementation of the interventions concerned, they were further considered to strengthen relationships among and between staff and students, which reciprocally translated with our a priori theme of eroding boundaries among and between these groups. In their high-quality description of the theory of change for Learning Together, Bonell et al. 165 explained:

[The intervention aims to] transform the whole-school climate and improve staff–student relationships, student commitment to learning and sense of belonging and participation in the school community.

Shinde et al. 181 similarly described:

The intervention’s conceptual framework … emphasises the importance of a positive school climate – i.e., supportive relationships between school community members, a sense of belonging to the school, a participative school environment, and student commitment to academic values.

Similarly, interventions that recruited parents or community members onto the groups leading implementation of interventions69,73,87,171,174,177,178,180,181,184,192,200 or that involved them in decision-making via other means169,197 represented efforts to involve the wider school community in school organisational management, enabling local tailoring and support for intervention activities, and the weakening of classification by eroding boundaries between the school and parents or other community members. This was exemplified in Flay et al.’s87 high-quality description of the theory of change for the AAYP school/community intervention:

The community program forged linkages among parents, schools, and local businesses. Each … school formed a local school task force consisting of school personnel, students, parents, community advocates, and project staff to implement the program components.

Reframing the instructional order

Interventions in this category commonly included elements aiming to modify schools’ teaching in some way to increase student engagement in learning. Theories of change differed as to whether this was limited to ensuring the inclusivity of teaching methods and intervention content, 87,177 or improving pedagogic practice or study skills,80,98,172,175,176,180,181,184,189 or whether teaching and learning were more radically reframed to weaken framing by providing students with a more active, collaborative role in constructing their own learning. 162–164,174,198

Below we consider these different forms of reframing of the instructional order and how they were theorised to erode boundaries within and beyond the school, the concept of boundary erosion being a key concept identified as an a priori theme in our coding framework, which reciprocally translated with concepts across many theories of change in this synthesis.

First, theories of change that referred to the importance of integrating culturally specific learning into school curricula described this as a means of recentring provision on the needs of students by including materials and methods that reflected the cultures and experiences of students or enabled them to appreciate those of others. 87,175–177 As explained by Flay et al. 87 in their account of the AAYP school/community intervention:87

Studies suggest that programs for African American youth should incorporate components that … enhance … cultural pride …. Hence, the interventions included the Nguzo Saba principles … which promote African American cultural values such as unity, self-determination, and responsibility.

Aligning with our a priori themes of eroding boundaries between school and the outside world, such approaches were explicitly theorised to erode boundaries between school culture and the diverse cultures of student families and communities, and to erode boundaries between students from different cultures. 87,162–164,177,198

Interventions including changes to pedagogy tended to focus on creating calmer, more respectful and equitable learning environments with better-quality teaching to improve engagement in learning, as well as greater accountability among teachers, as described by Devries et al. :80

The steps contain more than 60 different activities … focused around topics such as … creating a better learning environment, respect and understanding power relationships, improving teaching techniques, creating accountability.

In doing so, such approaches sought also to erode boundaries and improve relationships among students, and between students and teachers. As Kyriakides et al. 200 explained:

Effective teachers use different teaching strategies to keep different students involved in the classroom interactions which promote student learning and establish better relations among students and teachers.

Interventions improving pedagogic practices across a school also frequently organised teachers into teams to share good practice and social support. As well as being a mechanism to ensure effective whole-school change, this also reciprocally translated onto our a priori theme of eroding boundaries between previously individualistic teachers. Kyriakides et al. 178 described it thus:

[T]eachers interact on issues associated with learning and teaching… to create a business-like school and classroom environment … collaboration among teachers can only be beneficial, and could boost [school learning environment] quality … school management team… encourage… teaching staff to learn from each other by exchanging ideas and experiences on facing and reducing bullying.

Teaching study skills aimed to develop meta-learning skills over and above subject topic learning. 174,180,181,184 The cross-subject skills taught in the SEHER intervention, for example, included the following:

… time management, learning style, note taking, reading comprehension, memorisation techniques, and concentration techniques.

Shinde et al. 181

This aligned with our a priori theme of eroding boundaries between different academic subjects.

In one intervention,174 providing careers information and teaching job-seeking skills were theorised to introduce students to the world of work and to render instruction more meaningful to students, concepts that reciprocally translated onto our a priori theme of eroding boundaries between schools and the external world. As was explained by Gottfredson174 in her medium-quality description of the theory of change for Project PATHE:

The Career Exploration Programs, cosponsored by the program and a local technical college, provided high school students opportunities to participate in activities designed to introduce them to technical careers such as engineering, computer science, and industrial technology.

Turning now to the more radical reframings of the instructional order, authors referred to ‘constructivist’ approaches to learning as a means to reorient learning from more didactic approaches so that new knowledge was evaluated in the light of students’ own previous knowledge, skills and ideas. 162–164,174,185–188,198 Battistich et al. ,162 in their medium-quality description of the theory of change for the Child Development Project (CDP), explained it as follows:

Learning is inherently an active process in which students interpret new information in light of previous understandings and experiences, work through discrepancies, and construct new understandings … CDP’s instructional practices are consistent with this ‘constructivist’ approach to learning.

In such interventions, various forms of co-operative learning were used to reorient learning methods so that these moved away from individualistic tasks, assessments and rewards towards learning undertaken collaboratively by groups of students. 162–164,174,185,187,188,198 In the Cooperative Learning intervention,185–188 this involved a commitment to the principles of positive interdependence, whereby individual goals are aligned with the goals of the group such that individual success could be achieved only via group success, and competition and isolation among students are reduced. Via this approach, students were expected to:

… interact in ways that promote the goal attainment of others in the group, such as providing instrumental and emotional support, and sharing information and resources.

Van Ryzin and Roseth187

In the description of Cooperative Learning, the positive feelings that arise from such activities were theorised to result in a ‘benign spiral’188 that further increases such positive social interactions.

Such activities thus aimed to erode boundaries between students, as well as boundaries between academic education and students’ broader social and emotional development. As Battistich et al. 162 described:

[S]tudents reflect on and discuss their group interaction at the conclusion of the task. Thus, in addition to being an effective approach to learning, the co-operative activities also help students to build interpersonal bonds and develop social and ethical understanding and skills.

Van Ryzin and Roseth188 similarly noted:

We propose that this focus on social contact and the development of social skills enables co-operative learning to promote positive peer relations, which supports the development of empathy and, in turn, reduces bullying.

Constructivist co-operative approaches to learning also encouraged students from diverse backgrounds to draw on their own learning styles and experiences, as well as those of others, as a way to make sense and deepen their understanding of new information. As Battistich et al. 162 described:

The approach offers students of diverse cultural and ethnic backgrounds … to engage the curriculum in ways most appropriate to their own learning needs and interests; … meaningful interaction among students from different backgrounds and with diverse perspectives broadens the experiential base from which new and deeper understandings emerge.

By bringing diverse groups of students together, such approaches were theorised to foster insight and understanding of others’ perspectives to facilitate learning and ‘the breakdown of biases and prejudices among students who belong to different social groups’,186 thus encouraging the development of positive relationships and promoting social integration across peers.

As a result, these approaches were explicitly theorised to erode boundaries between school and the diverse cultures of student families and communities by making students’ backgrounds and experiences central to learning, and to erode boundaries between students from different backgrounds and cultures by enabling consideration and understanding of the perspectives of others.

Use of such approaches to learning also aimed to move from encouraging participation in learning via extrinsic rewards towards use of intrinsic rewards, whereby students become motivated by a more profound commitment to learning. 162–164,198 As Battistich et al. 162 described of the CDP:

A major emphasis of the CDP program is to tap into students’ intrinsic motivation to learn and… uphold the values of the school community. Teachers help students… understand that learning is both an inherently satisfying goal and a means to acquire the skills they need for successful, productive lives

Other theories of change emphasised the importance of ensuring that all students experienced academic success as an alternative way to engender students’ intrinsic commitment to learning and to school rules. 162–164,174,178,198 As described by Kyriakides et al. 178 in relation to the Dynamic Approach to School Improvement (DASI) intervention:

[P]erceived academic competence helps to prevent the development of norm-breaking behavior.

In the Gatehouse project, this was described as building a sense of ‘positive regard through valued participation in aspects of school’. 66

As further examples of more radical reframings of the instructional order, some interventions also included peer tutoring or mentoring, which was theorised to erode both boundaries between students (e.g. in different year groups) and boundaries in the role and status of teachers and students. 175,176,191,193,199 Mitchell193 in their lower-quality description of the theory of change for the PPP described it thus:

The Portland Peers Project utilized peer tutoring … [which] promoted student empowerment through involvement in meaningful activities, acquisition of information, and recognition of student responsibility … the project addressed … academic failure, lack of bonding to school, alienation, impaired confidence and self-esteem.

Some interventions also involved engaging parents in homework, to develop parents’ role in educating their children87,103,117,169,177,190,191,197,199 (eroding boundaries between the role of teachers and parents) or to draw on diverse family cultures in reframing the instructional order to erode school/home boundaries. 162–164,198 As described by Battistich et al. :162

The activities are designed to promote extended conversations and communication between students and their parents, connect the home to students’ experiences in school, and help students gain knowledge and understanding of their family’s beliefs, experiences, culture and heritage.

The approaches to reframing the instructional order outlined here were theorised to engender school commitment, as well as student knowledge and social support, reciprocally translating with concepts in the theory of human functioning and school organisation. This is considered in more detail in Engendering positive development and student commitment to reduce risk-taking behaviours.

Reframing the regulatory order

As with the instructional order, there was reciprocal translation of themes from included studies with our a priori themes whereby many theories engaged in some way with the concept of a school regulatory order and reframing this to engender greater student commitment. 69,71,73,88,162–168,194,195,198 As was described by Battistich et al. :162

Although often not explicitly recognized, schooling conveys important moral messages about how we should live our lives and how we should live together as people … The CDP program makes this ‘hidden curriculum’ … overt and supportive.

Theories of change again differed, as to whether reframing of the regulatory order was limited to staff modifying how they managed discipline or involved a more radical reframing. The former involved reorienting school discipline systems so that these better addressed students’ need for a safe and orderly environment, for example via enhanced classroom management, increased staff visibility or monitoring at break time or more consistent enforcement of rules and use of non-violent punishments. 80,98,171,172,180,181,184,192 More radically reframing gave students a more active role, via strategies such as rewriting school rules, learning-based discipline, restorative practice or learning social and emotional skills. 66–68,71,87,88,103,115,117,161–168,170,174,177,190,191,194,195,197–199 As with the split between interventions reframing the instructional order, some interventions included elements of both approaches. 171,178,189,192,200 The ways in which these approaches were theorised to work and how they reciprocally translated with our a priori theme of eroding boundaries within and beyond the school are outlined in School environment interventions developing student engagement in learning/social skills.

Enhanced classroom management, staff visibility at break time, more consistent enforcement of rules and use of non-violent punishments featured in several theories of change. 80,98,171,172,178,180,181,184,189,192,200 These aimed to promote a safer and more orderly school environment, but in some interventions were theorised to also have more direct impacts on violence or substance use by identifying and curtailing such activities, this diverging somewhat from our a priori themes. As Kyriakides et al. 200 described:

Increased monitoring of student behaviour during recesses and before the beginning of lessons can help school stakeholders to identify and intervene when bullying occurs. Teachers should be visible and vigilant in such common areas as hallways, stairwells, the canteen, the gym, and the crib.

In contrast, other approaches aimed to reframe the regulatory order by giving a more participative role to students. These were theorised to achieve impacts via mechanisms that reciprocally translated more closely with our a priori concepts of eroding boundaries within school. Several theories of change suggested that students’ contribution to writing or rewriting school rules or discipline policies with staff69,71,73,88,162–168,174,178,194,195,198,200 worked to erode boundaries between staff and students and among students, and would also increase overall student commitment to the resultant reframed regulatory order they took part in creating. As Kyriakides et al. 178 described:

The active involvement of teachers, students, and parents in defining the school policy on bullying and the strategies and action plans to face bullying may encourage their active participation in implementing these action plans for improvement purposes.

In effect, students became not merely subjects of the regulatory order, but its co-creators. In doing so, interventions aimed not merely to connect students to the regulatory order, but to reshape the regulatory order so that it was underpinned by democratic or ethical values, and aligned with students’ own values and cultures. 162–164,198 This aligned with our a priori concepts, in that this was another way in which interventions aimed to erode boundaries between schools and local communities.

Other theories of change suggested an even more fundamental reframing of the regulatory order that, in effect, eroded the boundary between schools’ regulatory and instructional orders. This constituted an elaboration of our a priori themes. These theories of change proposed a move from punitive discipline towards more learning-based discipline162–164,191,198,199 and the teaching of social and emotional skills66–68,71,87,88,103,115,117,162–171,177,190–192,194,195,197–199 and/or restorative practice71,88,161,165–168,171,192,194,195 whereby discipline became a focus for teaching, and conflict became an opportunity for learning. As described by Battistich et al. 164 in their lower-quality account of the theory of change for the CDP:

developmental discipline emphasizes a proactive ‘teaching’ approach to discipline rather than a coercive approach.

In Cross et al. ’s199 medium-quality description of the CFS theory of change:

[I]n contrast to promoting only punitive-based solutions to cyberbullying such as school suspension and withdrawing access to technology, which may exacerbate the problem, our formative research suggested … raising awareness among students, parents, and educators of the harms associated with the misuse of technology.

Similarly, social and emotional learning curricula aimed to teach students the skills to build strong relationships with peers and how to make healthy decisions, for example about violence and substance use. Cross et al. 171 described it thus in their medium-quality description of the theory of change for FSFF:

[C]lassroom level activities … were designed to complement students’ other social and emotional learning [and] focused on the reciprocal relationship between students who observe bullying, those who are bullied or bully others and their social environment.

Some curricula focused on increasing students’ involvement as active participants in the regulatory order by encouraging them to intervene as bystanders in incidents of bullying. 169,170

Restorative practice interventions also focused on building strong relationships between staff and students, and among students, to prevent conflict and misbehaviour, but also to repair such relationships in the aftermath of conflict. 71,88,161,165–168,171,192,194,195 Prevention occurred through activities such as ‘circle time’, in which students shared their feelings and developed ground rules for behaviour (resonating with other interventions in which students contributed to the formation of school rules and policies). As explained in Acosta et al. ’s161 high-quality description of the Restorative Practices Intervention’s theory of change:

restorative practices (e.g., circles, conferences) help… proactively build relationships with their peer and teachers and to have a more active voice in responding to specific school incidents. When students have positive interactions with their peers, this improves their peer relationships and can lead to more active participation in school.

Conflict was addressed through restorative conferences in which parties to a conflict met so that victims could describe the harm caused, perpetrators could take responsibility for this and work could be done to heal the relationship and identify appropriate reparations. These approaches aimed to erode boundaries between school members, as well as between schools’ regulatory and instructional orders. This aimed to prevent violence and other risk behaviours by ensuring that perpetrators did not become disconnected from the school community as a result of their experience of conflict and punitive discipline. Instead, perpetrators were reintegrated through a process of understanding the harm caused, recognising their responsibilities and making reparations. As was reported by Acosta et al. :161

[R]esponsive practices … ensure that offenders can take public responsibility for their behavior and reintegrate into normal community life.

Violence was prevented both by healing particular relationships and so curtailing further conflict, and by increasing student commitment to school and its norms of behaviour.

Other theories of change also suggested an erosion of the boundaries between the instructional and regulatory orders by aiming to broaden classroom teachers’ relationships with students from being merely educational towards including affective elements to root learning in a caring community. 69,71,73,88,162–168,194,195,198 As described by Battistich et al. :164

Teachers and administrators are encouraged to build warm, nurturing relationships with their students, and encourage students to develop warm, supportive, inclusive relationships with each other.

Some interventions further involved parents in discipline, encouraging parents to more closely monitor students’ behaviour and offering advice on how to do this. 103,117,169,171,190–192,197,199 The intended aims of such work reciprocally translate with our a priori themes of eroding boundaries between teacher and parental roles and eroding boundaries between the school and the family. As illustrated in Cross et al. ’s192 high-quality description of the FSFF programme’s theory of change:

The intervention was designed to… target parenting… identified as being protective of bullying behaviour … The family-level activities therefore worked in partnership with parents, building their awareness, attitudes and self-efficacy to role model, talk with and help their children to develop social competence and to prevent or respond to bullying.

Reframing relationships with the community

The involvement of parents and community members in decision-making groups was described previously as a means of reframing school organisational management and eroding boundaries between schools and the outside world. Several projects went further, aiming to build relationships between schools and local communities via various forms of voluntary work. Some projects encouraged students to volunteer in the local community,103,117,190,193 whereas others involved community members volunteering in schools and schools building alliances with community organisations. 87,103,117,177,190 All such components were theorised to develop school–community relations, a concept that reciprocally translates with our a priori concept of school–community boundary erosion. In a form of refutational synthesis, in one intervention the neighbourhood component was understood also to address community risk factors by building community cohesion and raising awareness of drug and alcohol issues,103,117,190 which diverged from our a priori themes suggested by the theory of human functioning and school organisation. 46

Engendering positive development and student commitment to reduce risk-taking behaviours

A key recurring theme across theories of change was that the abovementioned processes were theorised not merely to minimise risk behaviours, but to contribute towards students’ positive overall development. As emphasised by Battistich et al. :162

CDP’s emphasis is on the promotion of positive development … [to] promote children’s positive social, ethical, emotional, and intellectual development.

A subtheme was that interventions generally sought to achieve this via engendering student commitment to school (in terms of learning and belonging) by ensuring that school provision was reframed to meet students’ needs and align with their culture and values, and was built on positive relationships between members of the school community, as illustrated by this quotation from Bonell et al. :165

We theorised that … transforming the school environment to build student commitment to learning and sense of belonging in school … can be achieved by improving relationships between and among staff and students … and by better integrating students’ academic education and broader personal development.

Different theories used different terminology to refer to this, such as school attachment,67,68,115 bonding,163 school adjustment,169 inclusion115 and engagement and connection,67,68,161,170,184 but with clear reciprocal translation between these terms.

Some theories of change emphasised the importance of reframed provision being highly visible to students. This could consist of visible changes to school environment170,172 or highlighting changes to systems as a symbol of reframing provision based on student needs and informed by student participation. 166,194 Devries et al. 172 in their lower-quality account of the GST’s theory of change described it thus:

Some activities involve creating a better school environment by painting murals on school walls, and hanging codes of conduct in visible places.

In contrast, Bonell et al. 165 emphasised that:

Action groups are school meetings involving diverse students, and senior and junior staff … signal to the wider student body that the school cares about and intends to act on the views of staff and students to build a supportive school climate.

An example of refutational synthesis across theories of change was that there was a distinction between theories of change that viewed student commitment as reflecting a real investment in school activities and real commitment to shared values162–164,198 and theories of change that placed more emphasis on student commitment as outward compliance with rules. 178,200 For example, Solomon et al. 198 explained in their medium-quality description of the CDP’s theory of change how the intervention aimed to transform schools so that students shared their values:

We expect that students who have these needs met by experiencing a caring school community will feel strongly attached to the community, and that this attachment will lead them to feel personally committed to the values and goals the community promotes…

In contrast, the DASI intervention178 emphasised student compliance:

[S]chools could set up a motivation system to improve the school’s social environment by taking action to emphasize maintenance of the behavior code and the promotion of appropriate and positive behaviors outside the classroom.

In the middle were theories of change that viewed student commitment as reflecting students having a stake in the school institution so that they would be unwilling to engage in risk behaviours that might imperil their connection to school. 71,88,165–168,174,194,195 As Gottfredson174 explained:

[S]tudent involvement in constructive learning activities and consistent rewards for successful participation prevent delinquency by fostering attachment to school and giving students something to lose if they misbehave … youths who are ‘bonded’ to the social order have too much to lose by misbehavior.

Theories of change also described in what other ways intervention activities would benefit students’ positive development. These themes reciprocally translated with our a priori themes of promoting practical reasoning and affiliation, but elaborated these concepts. Several theories of change listed the different aspects of practical reasoning, such as student skills in reasoning, problem-solving, conceptual thinking, ethical and moral reasoning, and emotional learning, that would be engendered via intervention activities. 162–164,169,185–188,197,198 Theories of change also suggested impacts beyond cognitive skills to encompass self-efficacy, self-esteem, pro-social values, self-control, team work and perseverance. 87,175–177,185–188 Affiliation-related impacts included empathy and greater orientation to others, pro-social norms, as well as social, conflict resolution and communication skills. 71,87,165–169,175–177,185–188,195

Theories of change thus suggested that interventions would ultimately benefit students in terms of developing their autonomy, competence to make decisions and sense of belonging to place or people. 69,71,88,165–168,194,195 As summarised in Bonell et al. ’s166 high-quality description of Learning Together’s theory of change:

[T]he intervention aimed to enable young people to choose healthier behaviours by promoting their autonomy, motivation, and reasoning ability. These were to be promoted by increasing engagement with school.

Another example is given by Solomon et al. 198 in relation to the CDP:

[A]utonomy/influence refers to the individual’s opportunities to contribute to the group …; competence refers to the effectiveness and acceptance of the individual’s contributions to the group as well as his/her own academic and social efforts; and belonging is feeling that one is personally accepted by others ….

Across theories of change, attributes aligning with practical reasoning and affiliation were theorised to enable students to choose healthier, over riskier, behaviours.

Some theories of change also suggested that increases in student commitment to school might help reduce risk behaviours by mechanisms other than improving students’ affiliation and reasoning ability to choose healthier behaviours. 71,88,162–168,194,195,198 Battistich et al. 162 described it thus:

[T]he experience of being a valued member of a caring school community also should reduce the likelihood that students will seek to satisfy their need for social connection through association with gangs or other counter-cultural peer groups.

This theme reciprocally translated onto concepts previously developed by Jamal et al. 201 in a refinement of the theory of human functioning and school organisation,46 theorising that increasing student commitment to school will increase commitment to pro-school peer groups and decrease commitment to antischool groups and antisocial risk behaviours; our coding framework was elaborated to include this.

Van Ryzin and Roseth187 similarly described how intervention activities were expected to prevent deviant peer clustering to reduce health risk behaviours, but through a more direct mechanism than those already described:

[W]e attempted to interrupt the process of deviant peer clustering and reduce alcohol use by exposing youth to a broad cross section of their peers through collaborative, group-based learning activities in school … In this way, we hoped to interrupt the process of deviant peer clustering.

Such activities were also theorised to enable socially isolated students at risk of being bullied to develop new friendships and break down the process of homophily among bullies to reduce bullying. 186

Alignment with the theory of human functioning and school organisation

The theories of change of this group of two interventions aligned much less obviously with the theory of human functioning and school organisation. There was, however, some degree of reciprocal translation between the theories, as outlined below. A logic model summarising the theory of change for these interventions is provided in Figure 8.

FIGURE 8.

Logic model for school environment interventions developing student engagement in learning/social skills.

Primary intervention focus on social skills to increase school engagement and reduce risk behaviours

Each intervention was theorised to simultaneously increase students’ school engagement and reduce risk behaviours through activities primarily focused on a discrete social skills curriculum. Interventions did include whole-school elements, as per our inclusion criteria, but these were secondary to the curriculum. This contrasted with the mechanism described in the theory of human functioning and school organisation,46 in which reductions in risk behaviours are to be achieved by increasing commitment to school via changes to the school regulatory and instructional orders. 153

In the case of the Going Places programme,182,183 the main intervention component was a discrete classroom curriculum aiming to provide students with problem-solving, self-management, self-control, school involvement, communication and conflict resolution skills. As Simons-Morton et al. 182 described it:

[T]he primary component is a curriculum designed to influence perceptions, attitudes, and expectations about substance use and antisocial behavior and to improve self-efficacy and social skills and competence.

A secondary component encouraged all teachers to incorporate the use and rewarding of use of such skills in other lessons and across the school to reinforce curriculum learning; parents were also provided with materials to improve discipline and reinforce concepts from the curriculum at home.

The Positive Action intervention52,61,63,64,85,108–110,113,123,124,173,179 aimed to teach positive behaviours such as engaging in learning, having positive relationships with peers and avoiding antisocial behaviours), as well as the social skills needed to engage in these positive behaviours and how engaging in these activities would develop students’ sense of positive self-concept. In turn, this positive self-concept was theorised to be reinforced by students’ engagement in positive behaviours, and avoidance of risk behaviours including conflict and substance use. As reported in Flay et al. ’s85 high-quality description of the intervention’s theory of change:

The program teaches children what actions are positive, that they feel good when they do positive actions, and that they then have more positive thoughts and future actions.

The interventions aimed to develop skills in creative thinking, study, healthy habits, self-management, interpersonal and emotional management, self-honesty, goal-setting and decision-making to prevent engagement in violence and substance use. 85,123

Although the Positive Action intervention was informed by the theory of triadic influence, which includes environmental as well as individualistic mechanisms of reducing risk behaviour,202 and it did include whole-school actions, these were primarily focused on generalising (and rewarding) the application of skills developed in Positive Action lessons to other aspects of school, similar to the way in which the Going Places182,183 intervention aimed to generalise skills learnt through the curriculum.

There was some element of reciprocal translation of Positive Action’s theory of change with the theory of human functioning and school organisation46 in that it did aim to improve relationships among students and between staff and students, and promote students’ involvement in the local community through volunteering. This resonated with our a priori themes of eroding boundaries among students, between students and staff and between school and the outside world. However, these were theorised to occur primarily directly through the teaching of positive behaviours and the development of social skills achieved via the Positive Action curriculum, rather than through changes to the school regulatory or instructional orders, as theorised in the theory of human functioning and school organisation. 46

Although both interventions did aim to modify teaching to increase engagement in learning, the primary mechanism of reducing risk behaviours was a direct one, whereby teaching students new skills would reduce their involvement in bullying or substance use.

Intervention capability to be made sense of

A subtheme from studies of a diversity of interventions suggested that sense-making could be facilitated by an intervention’s ‘capability’ to be made sense of. This could be in terms of providing good-quality materials and ongoing support in the form of training, external facilitation, coaching or peer support. 66,69,71,88,113,167,189,193,194 In particular, materials and resources that included tangible, contextually relevant examples were reported to enable providers to better understand intervention activities and how they might use these in their setting, as was reported by one head teacher in Bonell et al. ’s71 evaluation of Learning Together of high reliability and usefulness:

The one thing schools need is a model, of how it’s going to work in the school, in a real-life school, so that they can almost touch it, taste it, feel it, and then start implementing it in their own schools.

In two studies, it was reported that staff were sometimes initially confused by intervention materials or external providers. 69,71 For example, in the study of the Healthy School Ethos (HSE) intervention, rated as being of high reliability and usefulness,69 an initial presentation by an external facilitator was reported to have caused staff and students to misunderstand the aims of a whole-school intervention:

‘Healthy School Ethos’ was … confused with the government’s ‘National Healthy Schools Programme’, and staff and students, including action-team members, initially assumed the focus was health education or healthy eating.

Bonell et al. 69

School capacity to make sense of an intervention

Another subtheme apparent in one UK study was66,69,71,167,189,190 that staff making sense of interventions could also be influenced by existing school priorities and capacity. 69,167,189,194 For example, those leading implementation in one school were said to have creatively reinterpreted Learning Together, an antibullying intervention, as an intervention aiming to maintain the emotional health of pressurised students in an academically selective school. 194 Warren et al. ’s194 evaluation of high reliability and usefulness reported:

One academically selective school did not view itself as having significant problems with bullying or aggression, and therefore used the [action group] primarily as a way to revise the homework policy.

Warren et al. 194 further reported that, in another school implementing Learning Together, the lead reinterpreted the staff–student action group as being a site for students to learn the skills needed to avoid or respond to bullying, rather than as a site for them to work with staff to co-ordinate intervention activities. This occurred in the context of low school capacity, in terms of the lead’s imprecise grasp of the intervention and inability to involve other staff in the group.

Intervention capability for local tailoring and adding value

A key subtheme apparent in several studies was that school staff assessed interventions in terms of ease of integration with existing school practices and interactions, which influenced whether or not they bought in to an intervention. 66,69,71,167,189,190 Interventions that could be locally tailored or had the potential to build on existing work were more likely to secure staff cognitive participation. Bosma et al. ’s190 evaluation, deemed to be of low reliability and usefulness, of the DARE Plus intervention described how the assessment phase of the intervention was essential to tailor the intervention to the school and hence develop the action team’s buy-in:

The assessment phase of the organizing process is critical to its long-term success. It is invaluable to take the required time to get to know the community before attempting to launch an action team.

Bonell et al. 71 describe how school staff bought in to use of restorative practice because this was viewed as providing a means of building on existing work and developing a consistent approach to discipline across the school:71

Although staff reported that some ‘restorative-type’ approaches were already being used in their school, this intervention was highly attractive because it could provide a new framework, process and resources for embedding restorative practices more consistently and more widely across the whole school.

Interventions not viewed as being capable of local tailoring or fitting with existing work often failed to engender staff buy-in, as reported by evaluations of the Responsive Classroom189 intervention (judged to be of high reliability and medium usefulness) and Positive Action113 intervention (judged to be of low reliability and usefulness). For example, O’Hare et al. ’s113 evaluation of Positive Action reported:

Some teachers and head teachers reported that they were reluctant to make whole-school changes. ‘In terms of ethos, it was a perfect fit. But a lot of what the programme does, we were already doing in ways that were slightly better.’.

A subtheme apparent in some studies was that this lack of intervention capability for tailoring or adding value was particularly undermining for whole-school elements, such as proposed changes to school policies or discipline systems. 113,189 School leaders could withhold buy-in when they felt that whole-school actions might jeopardise their wider strategies. This could be the case, for example, when interventions required changes to school rewards or discipline policies that school leaders thought might weaken the school’s ability to pass school inspections or attract parents to send their children to the school. As O’Hare et al. ’s113 evaluation of Positive Action reported:

Reluctance to change whole-school policy may be exacerbated by circumstances such as an upcoming … inspection: ‘It was hard to make a whole-school change to sanction and reward policy, so whole-school activity was harder to implement. [The government inspectorate] was coming and it would have been too big a change.’.

Intervention capability for using data to build buy-in

Another subtheme that recurred across the UK evaluations was that the provision of local data as part of the intervention could be useful in building buy-in. 66,69,71,88,167,190,194 The evaluation of the Learning Together intervention suggested that providing such data could make it harder for staff to dismiss the need for intervention. 71,167,194 A staff member on a pastoral team, quoted in Warren et al. ’s194 report on Learning Together, commented:

I remember when [facilitator] came to present to [senior leadership team] and said how terrible our data was … it was like a tumbleweed moment … it was a realistic … realisation for everyone if you know what I mean ….

However, in an example of refutational synthesis, several studies identified that the provision of data could sometimes undermine staff commitment when staff interpreted the data as a criticism of their work to date or when data did not indicate positive trends in the course of implementing an intervention. 69,88,167,195 Fletcher et al. ’s88 evaluation of the Learning Together intervention, deemed to be of high reliability and usefulness, reported thus:

The only negative issue… identified was that some… [senior managers] reported that the needs assessment felt too ‘negative’ at times, especially for school managers who had been in post for several years (and who likely saw this as a reflection on their many years of work and leadership at the school).

Similarly, Warren et al. 194 also reported on Learning Together:

Where year-on-year trends in such factors did not improve, staff members sometimes reported feeling dispirited. Such disappointments may have contributed to these schools not continuing intervention activities in the final year of the study.

Intervention capability in terms of provider credibility

The UK studies also suggested that staff were more likely to buy in to interventions when external providers appeared credible and had direct experience working in schools. 66,71,167 For example, in the case of the Learning Together intervention, school staff commitment to implementation was facilitated when the trainer was perceived as someone who understood and could speak to the contextual realities of what implementation might look like for staff in practice:

[T]he assistant head teacher at… [intervention school] suggested that the training would be more acceptable if it was delivered by: ‘Somebody who has established this in their school and can talk about nitty gritty things, like the systems, the processes, how they operate the structures on a day to day basis’.

Bonell et al. 71

Intervention capability in terms of student participation

A subtheme from several different UK evaluations of interventions aiming to encourage student participation in decisions was that students were more likely to buy in to an intervention when this offered an opportunity for them to take on active roles in decision-making or express their views. 69,71,113,167,194,195 Bonell et al.’s71 evaluation of Learning Together, for example, reported:

Key cross-cutting themes in student and staff accounts were that the focus on students ‘having a say’ and greater ‘respect’ for students’ views were important sources of acceptability… the action group was seen as a new ‘experience’ to be more ‘involved’ in ‘having a say’ at school…

The concept of ‘having a say’ reciprocally translated with the notion of having ‘their voice heard’ in the account of a staff member quoted in Warren et al. ’s195 evaluation, deemed to be of high reliability and usefulness:

I think… students will certainly enjoy… that we’re doing something like this so they can be involved in it and… can actually have their voice heard, that they can feel safe at school, that they can feel engaged with the teachers, that they can feel they’re listened to.

Such commitment from students, however, relied on the practice of genuine participation, which, in turn, relied on effective staff facilitation of such processes. When such processes were poorly facilitated and did not result in achieving visible action, this could result in disappointment and disengagement among students. 195

Staff potential for buy-in based on perceived need

Staff buy-in to interventions was reported to be influenced not only by the characteristics of the intervention but also by staff ‘potential’: whether they or the school were ready for such an intervention. A key subtheme here was that interventions should offer school leaders something they already knew they needed. 63,66,113,167,189 This might, for example, be a way of responding to government policies, pressures from parents or inspection requirements. Or it might address internal imperatives, such as school leaders’ existing strategies for school change. This theme was particularly clear in the UK studies of both HSE69 and Learning Together. 71,88,167,194 Fletcher et al. ’s88 pilot evaluation of Learning Together, for example, reported:

The [intervention] appeared… facilitated by features of the English secondary-school ‘market’ whereby parents have… choice over schools and… judge them… on… reputation and performance in ‘league tables’… [H]ead teachers and their [management teams] consistently reported that it was important to address aggressive behaviours … to recruit and retain ‘the best’ parents and students.

Thus, the intervention addressed an existing identified need to tackle aggression. Another example from the Learning Together evaluation concerned schools’ existing need to use needs data to improve school performance so that an intervention that provided this was attractive. From a study deemed to be of high reliability and usefulness, Bonell et al. 167 quote an intervention external facilitator:

[T]here’s an [English school inspectorate] agenda which drives all schools. And [the head] wanted to make sure as well that she was … they want ‘outstanding’, so she wants to make sure that she’s got all the information possible to make sure that they can get through that.

Similarly, interventions aiming to achieve whole-school change were reported to be more likely to get school leaders’ buy-in when there was already a recognition of the need for change, for example because of poor inspection results. 71,88 Reciprocally translating with this concept, it was apparent that, in schools where leaders perceived no such urgent imperative for change, genuine school buy-in was less likely. For example, Bonell et al.’s69 evaluation of the HSE intervention reported:

Progress was initially good at Hillside but was then observed to slow in term 2. Our facilitator thought this partly reflected … less sense of urgency in Hillside about using the project to improve the school.

Staff potential for buy-in based on existing strategies and values

A related subtheme was that school staff were reportedly more likely to buy in to a whole-school intervention when their existing strategies and values made this seem attractive. 63,71,88,110,167,189 For example, new head teachers were reported as particularly likely to buy in to interventions involving whole-school change because these aligned with their desire to make their mark and change schools. Fletcher et al. ’s88 evaluation of Learning Together reported:

[A] factor that appeared to be important at Goldstone Park was that the school had a new head teacher, which supported broader, structural changes to school environment to ensure that restorative practices could be implemented across the whole school.

Reciprocally translating with this concept of school leaders’ potential was teachers’ potential. 63,71 For example, teachers with a prior commitment to social and character education within their classes were more likely to implement curricula addressing this, according to Beets et al.’s63 study of Positive Action, deemed to be of medium reliability and low usefulness:

[T]eacher beliefs regarding their responsibility to teach [social and character development] concepts were significantly … related to their attitudes towards Positive Action …; attitudes towards the Positive Action program were positively related to the amount of the Positive Action curriculum delivered.

Similarly, school staff and students were also reported as more likely to commit to enacting restorative practices if staff already had some interest in this approach. A head teacher reported this in Bonell et al.’s71 evaluation of Learning Together:

…what attracted us was the idea that, you know, we could get something out of it… I mean, we’ve been trying to develop a kind of, more restorative approach to secure behaviour over a period of time [and] so this fitted in with the way we want to work.

In cases in which the values or priorities did not align, staff buy-in appeared less likely. 63,110,189 For example, when staff or students perceived restorative practice to be a softer option, they were reportedly unlikely to commit to enacting it, either rejecting it or implementing tokenistically. 189 As Anyon et al.’s189 study of Responsive Classroom reported:

In contrast, some middle school staff members’ beliefs about the value of punitive responses to problem behavior were incompatible with the core tenets of the intervention … These staff members believed that zero-tolerance policies … were more effective than RC [Responsive Classroom] approaches, which aim to enhance students’ intrinsic motivation to improve.

A subtheme among the UK studies concerned the possibility of schools committing to implementing only those intervention components that aligned with their existing strategies and values, rejecting or soft-pedalling components that they regarded as deviating from these. 69,167 Bonell et al.’s69 evaluation of the HSE, for example, reported:

[A]spects of each school’s ethos supported… the intervention while [others] were less supportive …. At Hillside… aspects of HSE were supported by the school’s… ethos of… involving students in decisions, while others such as the action on rewards may have been impeded by… limited baseline attention to engaging disaffected students.

Intervention capability as workable

A key subtheme was the importance of interventions being locally workable in staff enacting interventions as planned. 69,167,189 For example, curriculum materials that did not fit into the school curriculum or that did not provide staff with clear lesson plans or engaging materials tended to be adapted before they were delivered, or were not delivered at all. 66,167,194 Bonell et al.,167 for example, reported:

Many schools did not deliver the curriculum materials as they were packaged. Staff in several schools commented that that they needed to adapt the curriculum because it did not address the needs of their students, or because the materials were too simplistic.

Reciprocally translating with this general notion of workability was the notion of workability in terms of enactment in some settings more than others. The evaluation by Anyon et al. of the Responsive Classroom intervention reported that classroom elements were more workable in primary grade classrooms than in middle-school grade classrooms. 189 Teachers in the middle-school grade classrooms reported that the interventions could not be made to fit into their much more structured school day:

[T]he implementation of RC [Responsive Classroom] strategies took time away from teaching reading or math skills.

Anyon et al. 189

An important aspect of workability was the extent to which guidance materials spelt out how delivery should proceed. For example, materials underpinning restorative practice interventions needed to specify which staff members were responsible and whether the intervention was intended to complement or replace punitive discipline. 189

Some interventions were not collectively enacted as had been planned because their developers had not anticipated whether or not they would really work in the complex environment of schools. For example, Anyon et al.’s189 evaluation of the Responsive Classroom intervention found that a new approach to discipline failed to work in the reality of schools:

[P]articipants reported that a key RC [Responsive Classroom] strategy, Logical Consequences, in which a response to student misbehavior is tied to the specific incident and creates an opportunity for learning, was too unwieldy to implement in a way that students could anticipate and incorporate.

Planning groups as a key element of intervention capability

An important subtheme here was that interventions that included planning groups consisting of staff, sometimes also with involvement of students, parents or other community members, were reported to be particularly successful in ensuring collective action to enact interventions. This was apparent from reports of the Gatehouse Project intervention (of low reliability and medium usefulness), the Learning Together intervention (of high reliability and usefulness)66,69,71,167,194,195 and other interventions. 63,66,110,113,190,193

Diverse participation in such groups could support implementation by ensuring that the decisions made by the group were pragmatic and by achieving wider buy-in across the school community. Such groups were reported to be particularly facilitative of whole-school approaches. 66,69,194,195 Bond et al. ,66 in relation to the Gatehouse Project, reported that:

Because the make-up of the team was broadened to include administration and curriculum, not only has the work of the adolescent health team facilitated reviews of organizational structure, but it has also contributed to a substantial shift in the perceptions of what is the core business of schools.

Such groups could also help ensure that intervention activities added up to a co-ordinated process of integrated school transformation, rather than merely a disparate set of initiatives, for example as reported in Bond et al. ’s66 evaluation of the Gatehouse Project:

A consistent response from the key informants was that the Gatehouse Project offered a framework for coordinating such work and a set of lenses with which to review current strategies and implement new ones …. Supporting schools to make those links and to see the connections was … important.

Synergy between intervention components as a key element of intervention capability

A further notable subtheme was that some interventions had better synergies between intervention components than others, and this was important in enabling them to be implemented more successfully. 66,69,71,88,167,189,194,195 Synergy appeared to occur when the consequences of enacting one intervention activity helped facilitate the conditions required for the implementation of other components. This could ensure that priorities were identified, commitments built, and co-ordination and momentum created. In such cases, some intervention activities created the informational and relational resources needed to enable agents to enact other actions. For example, Bond et al. ’s66 evaluation of the Gatehouse Project reported the following:

It is clear from our work that these elements – the adolescent health team, the school social climate profile, and the critical friend – do not work in isolation …. The critical friend provides expertise, impetus, motivation, and links to external resources.

As described previously, data on student needs being provided as part of an intervention could encourage others to implement intervention activities. Data being provided as part of the intervention could lead to school staff producing or sharing other data. This, for example, was reported as an unexpected consequence of the sharing of student needs data in the UK studies of the Learning Together pilot:71,88

It was clear that all the schools in the pilot were very ‘data rich’ environments and this presentation of new, external student survey data to the action group triggered them to identify, analyse and triangulate multiple sources of data which they would not typically have used.

Fletcher et al. 88

External facilitation or coaching could similarly enable better implementation of other intervention activities. As was reported in Bond et al. ’s66 study of the Gatehouse Project:

School personnel identified the critical friend as an important aspect of the support network within the school, helping to keep up the momentum for the school-based team and developing shared understandings.

One obvious area of intervention synergy was where training components provided staff with the skills they needed to deliver other intervention elements. This could be valuable in ensuring staff accumulated and consolidated their skills, as was reported by a staff member in Anyon et al.’s + 208 study of the Responsive Classroom intervention:

One thing that was really helpful … is that I would use the language and then forget the language and so you’re coming back in and checking in with us in the [booster session] meetings, [which] really remind[s] me, ‘oh, yeah, I gotta do that’.

Some evaluation reports focused on the concept of synergy as facilitative of collective action, whereas other reports focused on lack of synergy as an inhibitor of collective action. For example, some evaluations reported that there was a noticeable lack of effective interaction between curriculum and whole-school components. In some cases, classroom curriculum activities were enacted, but whole-school changes intended to reinforce what students learned in these classes were incompletely delivered. 113 In another case, whole-school elements that aimed to build on existing school achievements were enacted, but curriculum elements were not delivered with fidelity because these were judged as unworkable by staff. 167,194

If intervention materials did not explain how components were intended to interact, this could also hinder collective action. Bonell et al.’s71 evaluation of Learning Together reported that:

[O]other staff reported ‘not knowing enough’ and that it was not always clearly communicated to them how it all ‘married up’.

School capacity to support collective action

The extent to which agents in schools could come together to collectively enact interventions also depended on the school resources available to these agents.

Intervention capability for reflexive monitoring

Processes of reflexive monitoring were best supported when interventions included this as an integral component. 66,69,88,167 Studies indicated that interventions were particularly successful when they included an action group that reviewed data, identified priorities, oversaw delivery and reflected on the results. Such processes enabled members to reflexively monitor what was being enacted and with what consequences. Evaluations suggested that this gave participants the permission and resources to try different things, persist with what was perceived as working and refine or reject what was perceived to go less well. This approach allowed staff to abandon activities viewed as unsuccessful without rejecting the intervention overall. For example, Bond et al.’s66 evaluation of the Gatehouse Project reported that:

This common purpose gave permission for teachers to try new strategies such as substantially restructuring student and teacher teams … [I]n one school, teachers worked together to reorganize classes into small groups of four or five learners and teachers into teaching teams to promote a collaborative and an academic environment.

These steps required staff to meet frequently to review information and monitor consequences. 190

As part of processes of reflexive monitoring, quick wins evidencing positive outcomes can help maintain and further build coalitions and commitment, and collective impetus to implement further intervention activities. Bosma et al.’s190 evaluation of DARE Plus, for example, reported that:

Teams with the highest numbers of activities were those that developed the most momentum, with each success leading to new ideas for working on issues.

As well as groups, ongoing support from training, facilitation or coaching could also support reflexive monitoring by providing an opportunity for reflection and/or an outsider perspective. The importance of an external facilitator was, for example, described as follows in Bond et al.’s66 evaluation of the Gatehouse Project:

The support that [critical friend] provided in the staff room, in staff meetings, has been invaluable. We wouldn’t be where we are now, because I’d never recognized the value of having a person who is not a practicing teacher in the school at the moment.

Collective reflexive monitoring to refine implementation

Reflexive monitoring could be a collective action oriented towards refining how an intervention was implemented. 66,193,194 For example, in the case of two interventions, over time, staff in some schools opted to recruit fewer disengaged or disadvantaged students to participate in intervention activities. 167,193 Mitchell193 reported of the PPP:

In 1989–90, school counsellors made a concerted effort to select at-risk, as well as able, students for the peer helper program … In 1990–91, counselors were more selective in choosing peer helpers and as a result, fewer high-risk students were selected as peer helpers.

When external facilitators were removed in the Learning Together intervention, this resulted in the overall fidelity of implementation declining, but some intervention components became mainstreamed so that their ‘form’ was modified at the same time as their ‘function’ became integrated within school policies and systems, as Warren et al.’s194 evaluation reported:

The absence of [facilitator] has been incredibly significant because she … was able to tie it in all the time to the agenda. And was a touchstone I suppose really for that. And then … so that … I think that was a loss.

Senior leadership team member, Warren et al. 194

Reflexive monitoring reinforcing implementation

A final subtheme that emerged from process evaluations was that reflexive monitoring could reinforce the conditions necessary for further implementation. 66,69,167,193 Staff and students in schools recognised through processes of reflexive monitoring that interventions had diverse consequences for different parts of the school systems, many of which were unanticipated. Instituting action teams with diverse membership could lead to staff with previously discrete roles coming to think more holistically about student development and well-being. For example, Bond et al. ’s66 evaluation of the Gatehouse Project reported that:

There was also evidence of changing professional identity – teachers shifted their position from being a teacher of a subject or program to placing the young person and learning at the center of practice.

Similarly, involving students in decision-making or being surveyed about their needs could transform staff and student attitudes by suggesting that the school was becoming a more participative institution. Bonell et al.’s69 evaluation of the HSE intervention reported:

I was speaking to some younger students … they said, ‘I can’t believe this is happening and we’re having a say in the rules …’. And they think, ‘if I make the rules, then I shouldn’t go against my own rules’.

Student action team member, Woodbridge, Bonell et al. 69

Sequence generation

Of the 18 trials included, 752,61,80,87,108,109,165–168,172,175–177,179–181,184,197 had adequate sequence generation and 1164,67,68,103,115,117,123,124,161,169–171,178,182,183,185–188,199,200 did not state how sequence generation was undertaken.

Allocation concealment

Of the 18 trials included, 1552,61,64,67,68,87,103,108,109,115,117,123,124,161,169–171,175–188,199,200 did not state how allocation concealment was undertaken. The remaining three80,165–168,172,197 trials provided information on how allocation was concealed, which was generally through the use of an independent statistician.

Blinding of participants, personnel and outcome assessors

Only one trial165–168 reported blinding of participants, study personnel and outcome assessors in regard to treatment allocation. A further 13 trials64,67,68,87,103,115,117,123,124,161,169,170,177,178,182,183,185–188,197,199,200 did not report information with which to make a judgement. Three trials52,61,80,108,109,172,175,176,179 did not include blinding of participants, personnel or outcome assessors. In one trial,180,181,184 participants and personnel were not blinded, but outcome assessors were.

Complete outcome data

Ten trials64,80,87,103,117,123,124,165–170,172,177,178,185–188,199,200 presented complete outcome data, whereas six trials52,61,67,68,108,109,115,171,175,176,179–181,184,197 did not and two trials161,182,183 were unclear. Specifically, one trial rated as not having complete outcome data52,61,108,109,179 did not have complete outcome data in the interim findings report,109 with subsequent reports from this trial rated as ‘unclear’ for completeness of outcome data. Complete outcome data were defined as balanced retention in trial arms with attrition of < 30%. Of the six trials without complete outcome data, three trials67,68,115,171,180,181,184 also had notably imbalanced attrition between arms.

No selective outcome reporting

Only one trial185–188 had evidence of selective outcome reporting, whereas six trials80,161,165–168,172,175,176,180,181,184,197 did not, with remaining trials at unclear risk of bias because of selective outcome reporting. For the one trial with evidence of selective outcome reporting, outcomes for bullying perpetration required further contact with authors to retrieve relevant data.

Accounted for clustering

Only one trial182,183 did not account for clustering at the school level.

Reduction of other sources of bias

Two trials80,170,172 did not reduce other sources of bias, whereas six trials reduced other sources of bias by using longitudinal analysis methods with baseline adjustment,87,165–168,177 using appropriate estimators to account for non-normal distributions,64,123,124,161,185–188 and multiple imputation. 197

Bias due to confounding

Of the five non-randomised evaluations included, two162–164,174,198 were rated as at critical risk of bias due to confounding, because of inappropriate control for clustering and insufficient control for confounding. Two85,173 were rated as at moderate risk of bias because confounding was expected, but appropriate confounding domains were included. One73 was rated as having a low risk of bias as confounding domains were included, results were adjusted for clustering and school allocation was quasi-random.

Bias in selection of participants for the study

One non-randomised evaluation162–164,198 was rated as being at serious risk of bias due to selection of participants for the study because the long-term follow-up163 reached a relatively small fraction of the full sample, with systematic differences between the original and follow-up samples. Two85,173 were rated as being at moderate risk of bias as start and follow-up of the intervention did not coincide for all participants. Two73,174 were rated as having a low risk of bias as analyses were prospective and follow-up coincided for all participants.

Bias in classification of interventions

Three non-randomised evaluations73,162–164,174,198 were rated as being at low risk of bias from classification of interventions. Two non-randomised evaluations85,173 were rated as being at moderate risk of bias because of how exposure was operationalised in these studies (e.g. as having implemented the intervention for > 4 years).

Bias due to deviations from intended interventions

All five evaluations were rated as being at low risk of bias due to deviations from intended interventions.

Bias due to missing data

Three evaluations73,85,173 were rated as being at low risk of bias due to missing data. One evaluation162–164,198 was rated as being at moderate risk of bias due to missing data, given high numbers of missing data and the lack of an analysis strategy to account for this. One evaluation73 did not present enough information to reach a judgement in this domain.

Bias in measurement of outcomes

Three evaluations73,162–164,174,198 were rated as being at moderate risk of bias in this domain because of the possibility of knowledge of assignment influencing outcome measurement. Two evaluations85,173 were rated as being at low risk of bias because of the use of administrative data for outcomes.

Bias in selection of the reported result

One evaluation73 was reported as being at low risk of bias in this domain. The remaining four85,162–164,173,174,198 were rated as being at moderate risk of bias because, although outcome reporting was reasonably complete, there was no point of comparison to confirm this.

Up to 1 year post baseline

Findings from three RCTs87,177,178,180,181,184,200 and one non-randomised evaluation73 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions on violence perpetration at up to 1 year post baseline (Figure 11). This was reflected in a meta-analysis suggesting an overall non-significant reduction in violence perpetration, but with significant heterogeneity between studies. A meta-analysis including four effect sizes from three studies87,177,178,180,181,184,200 suggested a non-significant impact in reducing the odds of violence perpetration at up to 1 year post baseline [odds ratio (OR) 0.83, 95% confidence interval (CI) 0.65 to 1.08]. However, this finding had substantial heterogeneity, with I² estimated at 62.7%.

FIGURE 11.

Effects on violence perpetration, up to 1 year post baseline, of interventions promoting student participation in school policy decisions.

More than 1 year post baseline

Findings from seven RCTs52,61,64,87,103,108,109,117,123,124,165–168,177,179–181,184,199 and one non-randomised evaluation162–164,198 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions on violence perpetration at > 1 year post baseline (Figure 12). This was reflected in a meta-analysis suggesting a non-significant and heterogeneous reduction in violence perpetration. A meta-analysis including 32 effect sizes from seven studies52,61,64,87,103,108,109,117,123,124,165–168,177,179–181,184,199 yielded a non-significant pooled estimate of interventions’ impact in reducing violence perpetration > 1 year post baseline (OR 0.69, 95% CI 0.47 to 1.03). This pooled effect was estimated with substantial heterogeneity, represented in an I² of 83.9%.

FIGURE 12.

Effects on violence perpetration, > 1 year post baseline, of interventions promoting student participation in school policy decisions.

Up to 1 year post baseline

Findings from two RCTs170,171 suggested possible, but inconsistent, impacts on violence perpetration of interventions promoting student relationships with teachers, but not student participation in school policy decisions, at up to 1 year post baseline (Figure 13). This was reflected in a meta-analysis suggesting a non-significant and moderately heterogeneous reduction in violence perpetration. A meta-analysis including nine effect sizes from two studies170,171 suggested a non-significant impact in reducing the odds of violence perpetration at up to 1 year post baseline (OR 0.77, 95% CI 0.14 to 4.21). This finding had a moderate amount of heterogeneity, with an I² of 37.1%. A sensitivity analysis model for this meta-analysis using fixed effects did not converge; thus, we could not compare robustness under different estimation methods.

FIGURE 13.

Effects on violence perpetration, up to 1 year post baseline, of interventions promoting student relationships with teachers, but not student participation in school policy decisions.

More than 1 year post baseline

Findings from two RCTs170,171 suggested unlikely and inconsistent impacts on violence perpetration of interventions promoting student relationships with teachers, but not student participation in school policy decisions, > 1 year post baseline (Figure 14). This was reflected in a meta-analysis suggesting a non-significant and moderately heterogeneous reduction in violence perpetration. A meta-analysis including nine effect sizes from two studies170,171 suggested a non-significant impact in reducing the odds of violence perpetration > 1 year post baseline (OR 0.88, 95% CI 0.10 to 7.68). This finding had a moderate amount of heterogeneity, with an I² of 56.2%. However, a sensitivity analysis using fixed effects suggested an impact of similar magnitude, but with borderline significance (OR 0.85, 95% CI 0.73 to 1.00). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.

FIGURE 14.

Effects on violence perpetration, > 1 year post baseline, of interventions promoting student relationships with teachers, but not student participation in school policy decisions.

Up to 1 year post baseline

Findings from two RCTs182,183,185–188 suggested that interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers, were unlikely to affect violence perpetration up to 1 year post baseline (Figure 15). This was reflected in a non-significant meta-analysis with minimal heterogeneity. A meta-analysis including two effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of violence perpetration at up to 1 year post baseline (OR 0.90, 95% CI 0.78 to 1.05). This finding did not include substantial heterogeneity, with an I² of 0%.

FIGURE 15.

Effects on violence perpetration, up to 1 year post baseline, of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers.

More than 1 year post baseline

Findings from two RCTs182,183,185–188 did not suggest clear impacts on violence perpetration of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers, > 1 year post baseline (Figure 16). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including two effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of violence perpetration > 1 year post baseline (OR 0.77, 95% CI 0.01 to 74.3). This finding included substantial heterogeneity, with an I² of 94.9%. A sensitivity analysis also did not suggest a possible significant impact of interventions (OR 0.78, 95% CI 0.47 to 1.28).

FIGURE 16.

Effects on violence perpetration, > 1 year post baseline, of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers.

Up to 1 year post baseline

At the end of the first school year of implementation of Friendly Schools,169 control group students (n = 863) and Friendly Schools students (n = 984) were not significantly different in their odds of bullying peers every few weeks (OR 0.66, 95% CI 0.39 to 1.13) or bullying peers at all (OR 1.15, 95% CI 0.81 to 1.63). This intervention targeted grade-4 students, who were 8–9 years of age at the trial’s start. ORs are expressed as the increase in odds of bullying in the control group, that is the first estimate, but not the second estimate, reflects a numerical benefit to the control group. Because only one RCT169 reported outcomes in this category, no meta-analysis was possible.

Up to 1 year post baseline

Findings from three RCTs103,117,178,180,181,184,200 suggested possible, but inconsistent, impacts on violence victimisation at up to 1 year post baseline of interventions promoting student participation in school policy decisions (Figure 20). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including nine effect sizes from three studies103,117,178,180,181,184,200 suggested a non-significant impact in reducing the odds of violence victimisation at up to 1 year post baseline (OR 0.79, 95% CI 0.42 to 1.08). This finding included substantial heterogeneity, with an I² of 92.8%.

FIGURE 20.

Effects on violence victimisation up to 1 year post baseline of interventions promoting student participation in school policy decisions.

More than 1 year post baseline

Findings from five RCTs80,103,117,165–168,172,180,181,184,199 and one non-randomised evaluation73 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions on violence victimisation > 1 year post baseline (Figure 21). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including 45 effect sizes from five studies80,103,117,165–168,172,180,181,184,199 suggested a non-significant impact in reducing the odds of violence victimisation at up to 1 year post baseline (OR 0.75, 95% CI 0.49 to 1.15). This finding included substantial heterogeneity, with an I² of 90.0%.

FIGURE 21.

Effects on violence victimisation > 1 year post baseline of interventions promoting student participation in school policy decisions.

Up to 1 year post baseline

Findings from three RCTs67,68,115,170,171 suggested possible, but inconsistent, impacts on violence victimisation at up to 1 year post baseline of interventions promoting student relationships with teachers, but not student participation in school policy decisions (Figure 22). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including 10 effect sizes from three studies67,68,115,170,171 suggested a non-significant impact in reducing the odds of violence victimisation at up to 1 year post baseline (OR 0.86, 95% CI 0.52 to 1.43). This finding included substantial heterogeneity, with an I² of 66.9%.

FIGURE 22.

Effects on violence victimisation up to 1 year post baseline of interventions promoting student relationships with teachers, but not student participation in school policy decisions.

More than 1 year post baseline

Findings from three RCTs67,68,115,170,171 suggested possible, but inconsistent, impacts on violence victimisation > 1 year post baseline of interventions promoting student relationships with teachers, but not student participation in school policy decisions (Figure 23). This was reflected in a non-significant and moderately heterogeneous meta-analysis. A meta-analysis including 14 effect sizes from three studies67,68,115,170,171 suggested a non-significant impact in reducing the odds of violence victimisation > 1 year post baseline (OR 0.89, 95% CI 0.73 to 1.07). This finding included moderate heterogeneity, with an I² of 35.0%.

FIGURE 23.

Effects on violence victimisation > 1 year post baseline of interventions promoting student relationships with teachers, but not student participation in school policy decisions.

Up to 1 year post baseline

At the end of the first school year of implementation (end of grade 4) of Friendly Schools,169 control group students (n = 863) and intervention group students (n = 984) were not significantly different in their odds of being bullied every few weeks (OR 1.16, 95% CI 0.87 to 1.54) or in their odds of being bullied at all (OR 1.49, 95% CI 1.14 to 1.94). ORs are expressed as the increase in odds of bullying in the control group; that is both estimates reflect a numerical benefit to the intervention group. Because only one RCT reported outcomes in this category, no meta-analysis was possible.

More than 1 year post baseline

Findings from two RCTs169,197 suggested that interventions promoting parent involvement only were unlikely to reduce violence victimisation > 1 year post baseline (Figure 24). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including 12 effect sizes from two studies169,197 suggested a non-significant impact in reducing the odds of violence perpetration > 1 year post baseline (OR 0.94, 95% CI 0.10 to 9.06). This finding included substantial heterogeneity, with an I² of 81.5%. A sensitivity analysis suggested a similar finding (OR 0.92, 95% CI 0.71 to 1.20).

FIGURE 24.

Effects on violence victimisation > 1 year post baseline of interventions promoting parent involvement only.

Up to 1 year post baseline

Findings from two RCTs103,117,180,181,184 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions in reducing substance use at up to 1 year post baseline. This was reflected in a non-significant and moderately heterogeneous meta-analysis overall, with similar findings for alcohol outcomes and tobacco outcomes separately, but meta-analyses with a small number of studies were sensitive to estimation method.

A meta-analysis including 16 effect sizes from two studies103,117,180,181,184 suggested a non-significant impact in reducing the odds of substance use at up to 1 year post baseline (OR 0.83, 95% CI 0.16 to 4.30). This finding included moderate heterogeneity, with an I² of 49.8% (Figure 27). However, a sensitivity analysis suggested a similar, but statistically significant, effect (OR 0.81, 95% CI 0.68 to 0.97). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.

FIGURE 27.

Effects on substance use up to 1 year post baseline of interventions promoting student participation in school policy decisions.

A separate analysis of alcohol outcomes drew on eight effect sizes from two studies103,117,180,181,184 (Figure 28). This suggested a non-significant impact in reducing the odds of alcohol use at up to 1 year post baseline (OR 0.87, 95% CI 0.29 to 2.61), with some heterogeneity (I² = 19.8%). However, a sensitivity analysis suggested a significant effect (OR 0.89, 95% CI 0.83 to 0.95).

FIGURE 28.

Effects on alcohol use up to 1 year post baseline of interventions promoting student participation in school policy decisions.

A separate analysis of smoking outcomes drew on six effect sizes from two studies103,117,180,181,184 (Figure 29). This suggested a non-significant impact in reducing the odds of smoking at up to 1 year post baseline (OR 0.81, 95% CI 0.09 to 7.26), with substantial heterogeneity (I² = 68.9%). However, a sensitivity analysis suggested a significant effect (OR 0.82, 95% CI 0.67 to 0.99).

FIGURE 29.

Effects on smoking up to 1 year post baseline of interventions promoting student participation in school policy decisions.

More than 1 year post baseline

Findings from seven RCTs52,61,64,87,103,108,109,117,123,124,165–168,175–177,179–181,184 and two non-randomised evaluations162–164,174,198 suggested possible, but inconsistent, impacts of interventions promoting student involvement in school policy decisions in reducing substance use > 1 year post baseline. This was reflected in a non-significant and substantially heterogeneous meta-analysis, with similar findings for alcohol outcomes and illicit drug use outcomes separately, and less evidence of impact on tobacco outcomes.

A separate analysis of omnibus substance use outcomes suggested a larger, but still substantially heterogeneous and non-significant, effect.

A meta-analysis including 55 effect sizes from seven studies52,61,64,87,103,108,109,117,123,124,165–168,175–177,179–181,184 suggested a non-significant impact in reducing the odds of substance use > 1 year post baseline (OR 0.81, 95% CI 0.57 to 1.15). This finding included substantial heterogeneity, with an I² of 78.1% (Figure 30).

FIGURE 30.

Effects on substance use > 1 year post baseline of interventions promoting student participation in school policy decisions.

A separate analysis of alcohol outcomes drew on 22 effect sizes from six studies52,64,117,166,175,180 (Figure 31). This suggested a non-significant impact in reducing the odds of alcohol use > 1 year post baseline (OR 0.81, 95% CI 0.56 to 1.18), with substantial heterogeneity (I² = 77.3%).

FIGURE 31.

Effects on alcohol use > 1 year post baseline of interventions promoting student participation in school policy decisions.

A separate analysis of smoking outcomes drew on 16 effect sizes from six studies (Figure 32). 52,64,117,166,175,180 This suggested a non-significant impact in reducing the odds of smoking > 1 year post baseline (OR 0.92, 95% CI 0.66 to 1.28), with substantial heterogeneity (I² = 78.8%).

FIGURE 32.

Effects on smoking > 1 year post baseline of interventions promoting student participation in school policy decisions.

A separate analysis of illicit drug use outcomes drew on 10 effect sizes from five studies52,64,166,175,180 (Figure 33). This suggested a non-significant impact in reducing the odds of illicit drug use > 1 year post baseline (OR 0.74, 95% CI 0.37 to 1.46), with substantial heterogeneity (I² = 84.3%).

FIGURE 33.

Effects on illicit drug use > 1 year post baseline of interventions promoting student participation in school policy decisions.

A separate analysis of omnibus substance use outcomes drew on seven effect sizes from three studies52,64,87 (Figure 34). This suggested a non-significant impact in reducing the odds of alcohol use at up to 1 year post baseline (OR 0.56, 95% CI 0.24 to 1.28), with moderate heterogeneity (I² = 48.1%).

FIGURE 34.

Effects on omnibus substance use outcomes > 1 year post baseline of interventions promoting student participation in school policy decisions.

Up to 1 year post baseline

In the original study cohort of the Gatehouse Project,67 findings were presented at up to 1 year post baseline for the odds of any drinking, regular drinking, binge drinking, any smoking, regular smoking and any cannabis use in the preceding 6 months. At the end of the first intervention year (end of year 8), the odds were not significantly different between the intervention and control groups for any drinking (OR 1.00, 95% CI 0.78 to 1.28), regular drinking (OR 1.09, 95% CI 0.77 to 1.57) or binge drinking (OR 0.95, 95% CI 0.69 to 1.32). Intervention students were not significantly less likely to smoke at all (OR 0.89, 95% CI 0.72 to 1.12), but they were less likely than control students to be regular smokers (OR 0.66, 95% CI 0.46 to 0.95). Differences in cannabis use were not significant (OR 0.98, 95% CI 0.69 to 1.40). The exact numbers of students participating in measurement waves were not available.

More than 1 year post baseline

Substance use findings in the original study cohort of the Gatehouse Project trial were further presented at the end of the second school year from baseline and at the end of the third school year from baseline, corresponding to the end of year 10. 67 Intervention students were not significantly different from control group students in the following odds: their odds of any drinking at the end of the second school year (OR 1.00, 95% CI 0.78 to 1.28) or at the end of the third school year (OR 0.96, 95% CI 0.69 to 1.33); their odds of being a regular drinker at the end of the second school year (OR 1.05, 95% CI 0.70 to 1.57) or at the end of the third school year (OR 1.13, 95% CI 0.77 to 1.66); or their odds of binge drinking at the end of the second school year (OR 0.99, 95% CI 0.70 to 1.38) or at the end of the third school year (OR 1.02, 95% CI 0.71 to 1.46). A similar pattern of null results was found for the odds of any smoking at the end of the second school year (OR 0.92, 95% CI 0.63 to 1.33) or at the end of the third school year (OR 0.91, 95% CI 0.67 to 1.24). Intervention students were not different from controls on any cannabis use in the previous 6 months at the end of the second school year (OR 1.06, 95% CI 0.75 to 1.49) or at the end of the third school year (OR 0.81, 95% CI 0.57 to 1.16). A subsequent analysis68 probed incident and prevalent cannabis use at the last follow-up. Prevalence of any cannabis use in the previous 6 months was not significantly different between the intervention (n = 1155) and control (n = 990) groups (OR 0.80, 95% CI 0.57 to 1.13), nor was incidence of any cannabis use in the previous 6 months different between intervention (n = 1062) and control (n = 941) groups (OR 0.81, 95% CI 0.56 to 1.15). Patterns were similar when comparing intervention and control groups on prevalence of weekly cannabis use (OR 0.74, 95% CI 0.45 to 1.20) and on incidence of weekly cannabis use (OR 0.73, 95% CI 0.43 to 1.25). Finally, the sequential cohort analysis115 used an omnibus substance use outcome to compare two different cohorts of year-8 students, in 1999 and 2001, after 2 years of exposure to the intervention. Neither the 1999 cohort, which drew on 1158 intervention group students and 1428 control group students (OR 0.84, 95% CI 0.61 to 1.18), nor the 2001 cohort, which drew on 966 intervention group students and 1497 control group students (OR 0.85, 95% CI 0.65 to 1.12), demonstrated a significant impact of the Gatehouse Project intervention in reducing substance use.

Up to 1 year post baseline

Findings from two RCTs182,183,185–188 suggested possible, but inconsistent, impacts on reducing substance use of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers, at up to 1 year post baseline (Figure 35). This was reflected in a non-significant and substantially heterogeneous meta-analysis, but meta-analyses with a small number of studies were sensitive to estimation method.

FIGURE 35.

Effects on substance use up to 1 year post baseline of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers.

A meta-analysis including three effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of substance use at up to 1 year post baseline (OR 0.76, 95% CI 0.20 to 2.86). This finding included substantial heterogeneity, with an I² of 66.2%. However, a sensitivity analysis suggested a significant impact of similar magnitude (OR 0.80, 95% CI 0.67 to 0.95). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.

A separate meta-analysis of alcohol outcomes drew on two effect sizes from two studies182,183,185–188 (Figure 36). This suggested a non-significant impact on reducing alcohol use at up to 1 year post baseline (OR 0.82, 95% CI 0.59 to 1.15), with substantial heterogeneity (I² = 66.0%).

FIGURE 36.

Effects on alcohol use up to 1 year post baseline of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers.

More than 1 year post baseline

Findings from two RCTs182,183,185–188 suggested possible, but inconsistent, impacts on reducing substance use of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers > 1 year post baseline (Figure 37). This was reflected in a non-significant and substantially heterogeneous meta-analysis.

FIGURE 37.

Effects on substance use > 1 year post baseline of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers.

A meta-analysis that included eight effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of substance use at up to 1 year post baseline (OR 0.67, 95% CI 0.04 to 10.30). This finding included substantial heterogeneity, with an I² of 84.5%. However, a sensitivity analysis suggested a finding of similar magnitude, albeit statistically significant (OR 0.68, 95% CI 0.51 to 0.91). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.

A separate meta-analysis of alcohol outcomes drew on five effect sizes from two studies182,183,185–188 (Figure 38). This suggested a non-significant impact on reducing alcohol use > 1 year post baseline (OR 0.70, 95% CI 0.02 to 21.89), with substantial heterogeneity (I² = 87.3%). A sensitivity analysis generated a more precise CI, but did not change the conclusion (OR 0.70, 95% CI 0.48 to 1.03).

FIGURE 38.

Effects on alcohol use > 1 year post baseline of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers.

Across all substance use outcomes

Overall, meta-analyses suggested that interventions promoting commitment to school generate small, but statistically significant, impacts, and, potentially, impacts significant to public health, on reducing substance use at up to 1 year post baseline (OR 0.83, 95% CI 0.70 to 0.97). This analysis drew on 25 effect sizes from five studies67,117,181,182,186 and included a moderate amount of heterogeneity, with an I² of 54.0% (Figure 39). Interventions promoting commitment to school also generate small, but statistically significant, impacts, and, potentially, impacts significant to public health, on reducing substance use > 1 year post baseline (OR 0.79, 95% CI 0.62 to 0.998). This analysis drew on 81 effect sizes from 10 studies64,67,68,87,108,109,115,117,166,168,175,180,182,185 and included a substantial amount of heterogeneity, with an I² of 76.9% (Figure 40).

FIGURE 39.

Effects on substance use up to 1 year post baseline of all interventions.

FIGURE 40.

Effects on substance use > 1 year post baseline of all interventions.

Alcohol outcomes

Alcohol use outcomes were meta-analysed for up to 1 year post baseline and for > 1 year post baseline. An analysis drawing on 14 effect sizes from five studies67,117,181,182,186 (Figure 41) suggested a small, and statistically non-significant, reduction in alcohol use, caused by interventions promoting commitment to school, up to 1 year post baseline (OR 0.88, 95% CI 0.74 to 1.04). This finding had a moderate amount of heterogeneity (I² = 42.5%). An analysis drawing on 35 effect sizes from nine studies52,64,67,117,166,175,180,182,186 (Figure 42) suggested a small, and statistically non-significant, reduction in alcohol use, caused by interventions promoting commitment to school, > 1 year post baseline (OR 0.81, 95% CI 0.63 to 1.03). This finding had substantial heterogeneity (I² = 75.6%).

FIGURE 41.

Effects on alcohol use up to 1 year post baseline of all interventions.

FIGURE 42.

Effects on alcohol use > 1 year post baseline of all interventions.

Smoking outcomes

Smoking outcomes were meta-analysed for up to 1 year post baseline and for > 1 year post baseline. An analysis drawing on eight effect sizes from four studies67,117,181,182 (Figure 43) suggested a statistically non-significant reduction in smoking, caused by interventions promoting commitment to school, up to 1 year post baseline (OR 0.77, 95% CI 0.54 to 1.09). This finding had substantial heterogeneity (I² = 70.7%). An analysis drawing on 21 effect sizes from eight studies52,64,67,117,166,168,175,180,182 (Figure 44) suggested a statistically non-significant reduction in smoking, caused by interventions promoting commitment to school, > 1 year post baseline (OR 0.87, 95% CI 0.69 to 1.10). This finding had substantial heterogeneity (I² = 77.3%).

FIGURE 43.

Effects on smoking up to 1 year post baseline of all interventions.

FIGURE 44.

Effects on smoking > 1 year post baseline of all interventions.

Illicit drug use

Illicit drug use outcomes were meta-analysed for up to 1 year post baseline and for > 1 year post baseline. An analysis drawing on three effect sizes from two67,181 studies (Figure 45) suggested a small, and statistically non-significant, reduction in illicit drug use, caused by interventions promoting commitment to school, up to 1 year post baseline (OR 0.89, 95% CI 0.16 to 5.01). This finding had a low amount of heterogeneity (I² = 3.1%) and was imprecisely estimated. A sensitivity analysis suggested a similar conclusion as to the magnitude and significance of the effect (OR 0.85, 95% CI 0.66 to 1.08). An analysis drawing on 16 effect sizes from six studies52,64,67,68,166,175,180 (Figure 46) suggested a statistically non-significant reduction in illicit drug use, caused by interventions promoting commitment to school, > 1 year post baseline (OR 0.76, 95% CI 0.46 to 1.24). This finding had substantial heterogeneity (I² = 80.7%).

FIGURE 45.

Effects on illicit drug use up to 1 year post baseline of all interventions.

FIGURE 46.

Effects on illicit drug use > 1 year post baseline of all interventions.

Omnibus substance use outcomes

Omnibus substance use outcomes were meta-analysed for > 1 year post baseline only (Figure 47). An analysis drawing on nine effect sizes from four studies52,64,87,109,115 suggested a statistically non-significant reduction in general substance use > 1 year post baseline (OR 0.67, 95% CI 0.32 to 1.42). This finding had substantial heterogeneity (I² = 63.3%).

FIGURE 47.

Effects on omnibus substance use outcomes > 1 year post baseline of all interventions.

Age

Of the relevant studies, two RCTs report moderation or subgroup analyses by age. 171,177 Evidence for moderation by age on violence perpetration was sparse and inconsistent.

The RCT of the AAYP school/community intervention177 used a longitudinal model to examine moderation of intervention impact on violence perpetration by age over 4 school years. Although the main effect of the school and community intervention in the AAYP was a significantly reduced rate of increase in violence overall, compared with the control intervention (β = –0.155, SE 0.051; p = 0.002), the interaction term for intervention and age (β = –0.150, SE 0.056; p = 0.007) suggested an even stronger effect among older students. In practical terms, children 1 standard deviation below the mean (i.e. aged 9.79 years) did not experience effects in reducing the rate of violence increase over time (β = 0.045, SE 0.105; p = 0.66), but children 1 standard deviation above the mean (i.e. aged 12.47 years) did experience effects (β = –0.356, SE 0.074; p < 0.001). The RCT of FSFF171 stratified analysis into children in grade 4 and children in grade 6 and compared them on bullying perpetration outcomes at the end of the first and second school years from baseline. These effects are presented in depth in Appendix 7. Analysis included comparisons of both medium-intensity and low-intensity interventions against high-intensity interventions on outcomes of bullied others versus not and bullied others frequently versus not. No formal interaction tests were presented; however, a pattern of non-significant effects and inconsistent patterning of differential effects between grade-4 and grade-6 students across outcomes did not suggest any evidence of moderation by age in this trial.

Sex

Of the relevant studies, six RCTs report moderation or subgroup analyses by sex. 64,87,103,108,166,168,181 Evidence for moderation by sex of violence perpetration was inconsistent, with some studies87,103,108,166,168 suggesting patterns of differential benefits favouring boys and others108,181 suggesting patterns of differential benefits favouring girls (Figure 48). Two studies87,103 did not report formal interaction tests, which made drawing conclusions on this body of evidence more challenging.

FIGURE 48.

Moderation by sex. a, Individual bars represent studies, with dark shading representing a statistical test of moderation and light shading representing an informal description of moderation. Full-height bars represent consistently significant evidence of moderation; bars of three-quarters height represent evidence of mixed significance, and half height represents non-significant evidence of moderation.

In the AAYP school/community intervention RCT,87 estimates for growth in violence over 4 school years were presented stratified by sex. A formal interaction test was not presented; however, boys in the school and community intervention group experienced decreased growth in violent behaviours, compared with the control group, with a relative reduction by the end of grade 8 of 47% (p = 0.02). However, girls in the intervention group experienced a non-significant increase in violent behaviours, compared with girls in the control group.

Similarly, results for DARE Plus103 were stratified by sex, but without a formal interaction test. At the end of the second school year from baseline, boys receiving DARE Plus103 had a significant decrease in physical violence, compared with boys in the control group (M = 3.58 vs. M = 4.23; p = 0.03), whereas girls had a non-significant increase (M = 2.05 vs. M = 2.14; p = 0.64). Boys had a marginally significant decrease in verbal violence (M = 6.44 vs. M = 7.12; p = 0.06), whereas girls had a non-significant decrease (M = 5.77 vs. M = 5.92; p = 0.65), and neither boys (M = 1.70 vs. M = 1.97; p = 0.24) nor girls (M = 0.43 vs. M = 0.34; p = 0.56) were significantly different on weapon-carrying. This suggests some, albeit tenuous, evidence that DARE Plus produces favourable effects for boys.

Learning Together168 compared intervention effects at 36 months among boys and girls on an overall score of aggression, as measured by the Edinburgh Study of Youth Transitions and Crime (ESYTC) scale, and the modified aggression subscale score. There was some suggestion of a greater effect among boys than girls on the ESYTC scale score [boys: β = –0.33 (95% CI –0.73 to 0.06); girls: β = 0.04 (95% CI –0.32 to 0.39)], with marginal significance for the interaction term (p = 0.0890). However, there was significant evidence of interaction (p = 0.0029) on the modified aggression subscale score [boys: β = –0.53 (95% CI –0.89 to –0.18); girls: β = –0.03 (95% CI –0.37 to 0.31)]. A subsequent analysis168 examined moderation in perpetration of cyberbullying and wider aggression at both 24 and 36 months post baseline. There was no evidence of moderation on wider aggression at either time point or on cyberbullying perpetration at 36 months; however, at 24 months, the differential impact of the intervention on cyberbullying perpetration was significant (p = 0.002) for boys (OR 0.61, 95% CI 0.41 to 0.89), compared with girls (OR 1.19, 95% CI 0.85 to 1.67).

The Chicago trial of Positive Action108 considered moderation by sex for both violence-related behaviours and bullying behaviours using a longitudinal model over 6 school years. No significant moderation was reported for violence-related behaviours; however, the intervention had a larger impact by the study’s end in reducing violence among girls (d = –0.51) than among boys (d = –0.23), as suggested by significant three-way interactions between intervention, time and sex.

The Hawaii trial of Positive Action64 presented violence perpetration outcomes for boys and girls both as individual behaviours and as a summary score; however, only the summary score included a formal test of interaction. Individual violent behaviours did not show a clear pattern of greater effect for boys or girls. Although carrying a knife (boys: OR 0.27; girls: OR 0.51) and threatening to cut or stab someone (boys: OR 0.30; girls: OR 0.48) both reflected numerically greater intervention benefits for boys, the opposite was the case for cutting or stabbing someone (boys: OR 0.29; girls: OR 0.25), carrying a gun (boys: OR 0.33; girls: OR 0.58) or shooting at someone (boys: OR 0.14; girls: OR 0.22). The overall interaction test on the sum score of violent behaviours suggested a greater, but non-significant, intervention impact in reducing violent behaviours among boys than girls [interaction incidence rate ratio (IRR) 0.67, 90% CI 0.35 to 1.28; p = 0.158].

Finally, in the trial of SEHER,181 violence perpetration was examined at 8 months post baseline. In the lay counsellor SEHER ‘Mitra’ (LSM) (leader) group, compared with the control group, girls had a greater reduction in perpetration than boys [girls: OR 0.59 (95% CI 0.33 to 1.04); boys: OR 0.81 (95% CI 0.53 to 1.21)]. This yielded a significant interaction effect (p = 0.006). In contrast, there was no significant difference between boys and girls (p = 0.444) in the teacher SEHER Mitra (TSM) group, compared with the control group [girls: OR 1.61 (95% CI 0.91 to 2.85); boys: OR 1.51 (95% CI 1.00 to 2.29)].

Socioeconomic position

Only one trial (Learning Together)166,168 considered moderation by socioeconomic position for violence perpetration. The main analysis did not find evidence of moderation by socioeconomic position at 36 months post baseline, with intervention effects favouring students with low socioeconomic position on the ESYTC scale [low: MD –0.32 (95% CI –0.78 to 0.14); high: MD 0.12 (95% CI –0.24 to 0.49)], but without significant interaction (p = 0.0750), and intervention effects favouring students with high socioeconomic position on the modified aggression subscale [low: MD –0.04 (95% CI –0.43 to 0.36); high: MD –0.27 (95% CI –0.61 to 0.07)], but again without significant interaction (p = 0.2176). Family affluence was not a moderator of cyberbullying perpetration or wider forms of aggression at 24 or 36 months post baseline. 168

Baseline risk

Two trials, Cooperative Learning186 and Learning Together,166 considered moderation by baseline risk for violence perpetration. In Cooperative Learning,186 baseline risk was operationalised as student engagement with school. At about 5.5 months post baseline, a significant interaction between intervention condition and student engagement was found for bullying perpetration (β = 0.09, SE 0.04; p < 0.05), suggesting that the intervention was more effective at reducing violence perpetration among those with low baseline levels of school engagement.

In Learning Together,166 baseline risk was operationalised as experience of bullying victimisation, measured by the Gatehouse Bullying Scale (GBS), and aggression, measured by the ESYTC scale. Although there was a numerical trend towards greater intervention effect among students with high levels of baseline bullying victimisation, this was not significant at 36 months (p = 0.4422) for the overall ESYTC scale score [low levels of victimisation: MD 0.02 (95% CI –0.34 to 0.37); high levels of victimisation: MD –0.17 (95% CI –0.63 to 0.29)], nor was it significant (p = 0.8100) for the modified aggression subscale [low levels of victimisation: MD –0.18 (95% CI –0.52 to 0.17); high levels of victimisation: MD –0.22 (95% CI –0.62 to 0.18)]. However, baseline aggression significantly moderated intervention effects on the ESYTC scale at 36 months (p < 0.0001), with evidence of an intervention-generated increase in aggression among students with low baseline aggression (MD 0.71, 95% CI 0.33 to 1.10), and an intervention-generated decrease in aggression among students with high baseline aggression (MD –0.65, 95% CI –1.03 to –0.27). This moderation was in the same direction for the modified aggression subscale score, albeit not statistically significant (p = 0.0933), with no evidence of impact on students with low baseline aggression (MD 0.03, 95% CI –0.33 to 0.39), and numerical, but not statistical, impact on students with high baseline aggression (MD –0.25, 95% CI –0.60 to 0.10).

Age

Two studies reported subgroup analyses of violence victimisation outcomes by age: one RCT171 and one non-RCT. 163 Although neither study presented a formal interaction test, there was little evidence of moderation by age.

The RCT of FSFF171 stratified analysis into children in grade 4 and children in grade 6 and compared them on bullying victimisation outcomes at the end of the first and second school years from baseline. These effects are presented in depth in the narrative synthesis in Appendix 7. Analysis included comparisons of both medium-intensity and low-intensity interventions against high-intensity intervention on outcomes of bullied others versus not, and bullied others frequently versus not, and demonstrated relatively few significant effects either by intervention or within year group. No formal interaction tests were presented; however, inconsistent patterning of differential effects between grade-4 and grade-6 students across outcomes did not suggest any evidence of moderation by age in this trial. An analysis of the effectiveness of the CDP163 on a measure of violence victimisation stratified children by grades 6, 7 and 8. No formal interaction test was presented, but the magnitude of differences was relatively similar between children in grades 6 (intervention M = 1.88, control M = 2.00), 7 (intervention M = 1.88, control M = 1.89) and 8 (intervention M = 1.78, control M = 1.184), with an overall effect that was marginally significant (p < 0.10). This suggests little evidence of moderation by age.

Sex

Four RCTs presented moderation or subgroup analyses by sex for violence victimisation outcomes. 80,103,117,166,168,172,181 Evidence was highly inconsistent across studies. Although the DARE Plus trial and peer violence outcomes from the GST trial suggested greater intervention impacts for boys, findings from the SEHER RCT indicated a pattern of greater impacts for girls. Findings from the Learning Together trial did not provide evidence of moderation favouring either boys or girls (see Figure 48).

In the main report of the DARE Plus RCT,117 physical victimisation was analysed via longitudinal model and stratified by sex, although no formal interaction test was presented. Among boys, DARE Plus generated a significant (one-tailed p = 0.02) decrease in the rate of growth of physical victimisation (β = –0.10 vs. β = 0.03), whereas rates of growth among girls were virtually identical between conditions (both β = 0.00, one-tailed p = 0.45). In a subsequent analysis,103 differences in overall victimisation at the end of the second school year from baseline were not significant between the DARE Plus and control (p = 0.11) groups for boys (M = 7.99 vs. M = 8.62), nor were they significant for girls (M = 5.16 vs. M = 5.33; p = 0.62).

In the main report of the GST RCT,172 the primary outcome was physical violence from school staff in the previous week, as reported by students. The intervention had a significantly greater impact (p = 0.043) among boys (OR 0.34, 95% CI 0.21 to 0.56) than girls (OR 0.46, 95% CI 0.29 to 0.74). A subsequent report80 considered violence outcomes separately and included peer violence outcomes. Summary analyses of violence, including emotional, physical and sexual violence, suggested that the intervention had a greater effect among boys than among girls for violence from staff or peers in the previous week [boys: OR 0.34 (95% CI 0.22 to 0.53) vs. girls: OR 0.55 (95% CI 0.36 to 0.84); p < 0.0001] or the previous term [boys: OR 0.20 (95% CI 0.11 to 0.37) vs. girls: OR 0.44 (95% CI 0.24 to 0.82); p < 0.0001]. Similar patterns were found for any staff violence in the previous week [boys: OR 0.34 (95% CI 0.21 to 0.54) vs. girls: OR 0.49 (95% CI 0.31 to 0.77); p = 0.009] or previous term [boys: OR 0.21 (95% CI 0.12 to 0.37) vs. girls: OR 0.45 (95% CI 0.26 to 0.80); p < 0.0001]. Differences between boys and girls for emotional violence from staff were in similar directions, but not significant for either the previous week [boys: OR 0.68 (95% CI 0.40 to 1.16) vs. girls: OR 0.85 (95% CI 0.52 to 1.40); p = 0.354] or the previous term [boys: OR 0.60 (95% CI 0.40 to 9.92) vs. girls: OR 0.76 (95% CI 0.50 to 1.14); p = 0.218]. As in the main report, which included physical violence in the previous week, differences in staff violence in the previous term were significantly different (p < 0.0001) for boys (OR 0.20, 95% CI 0.11 to 0.36), compared with girls (OR 0.44, 95% CI 0.25 to 0.78). However, restricting to severe physical violence did not yield a similar or significant pattern of results, for either the previous week [boys: OR 0.77 (95% CI 0.19 to 3.13) vs. OR 0.31 (95% CI 0.11 to 0.91); p = 0.227] or the previous term [boys: OR 0.38 (95% CI 0.16 to 0.86) vs. OR 0.70 (95% CI 0.34 to 1.45); p = 0.174]. The impacts on sexual violence from school staff towards students in the previous term were not different (p = 0.648) between boys (OR 0.85, 95% CI 0.27 to 0.70) and girls (OR 1.20, 95% CI 0.46 to 3.10). Differences in intervention effect on peer violence were not significant for any form of violence, for either the previous week [boys: OR 0.62 (95% CI 0.43 to 0.89) vs. girls: OR 0.77 (95% CI 0.54 to 1.09); p = 0.192] or the previous term [boys: OR 0.61 (95% CI 0.43 to 0.88) vs. girls: OR 0.74 (95% CI 0.52 to 1.05); p = 0.184]. When stratified by form of violence, effects were very similar for emotional violence, in both the previous week [boys: OR 0.59 (95% CI 0.42 to 0.83) vs. girls: OR 0.77 (95% CI 0.55 to 1.08); p = 0.731] and the previous term [boys: OR 0.67 (95% CI 0.48 to 0.95) vs. girls: OR 0.68 (95% CI 0.48 to 0.95); p = 0.964]. There was similarly little evidence of difference in physical violence in the previous week [boys: OR 0.82 (95% CI 0.51 to 1.30) vs. girls: OR 0.72 (95% CI 0.48 to 1.12)] or in the previous term [boys: OR 0.63 (95% CI 0.44 to 0.90) vs. girls: OR 0.88 (95% CI 0.62 to 1.26); p = 0.064]. However, although relying on few cases, there was some marginally significant signal of differential impact on sexual violence in the previous week [boys: OR 0.38 (95% CI 0.07 to 1.96) vs. girls; OR 2.33 (95% CI 0.73 to 7.51); p = 0.061] and in the previous term [boys: OR 0.64 (95% CI 0.15 to 2.72) vs. girls: OR 3.39 (95% CI 1.22 to 9.39); p = 0.056].

In the main report of Learning Together,166 GBS scores at 36 months did not suggest a difference (p = 0.6113) in intervention effect between boys (MD –0.04, 95% CI –0.08 to 0.001) and girls (MD –0.03, 95% CI –0.06 to 0.01). Subsequent analyses168 did not find that sex moderated cyberbullying victimisation at 24 or 36 months.

Finally, in the trial of SEHER,181 violence victimisation, as measured by frequency of bullying, any victimisation and experience of forced sex, was examined at 8 months post baseline. In the LSM group, both boys (d = –0.39, 95% CI –0.53 to –0.25) and girls (d = –0.51, 95% CI –0.76 to –0.26) experienced a lower frequency of bullying, but the numerically greater impact for girls was not statistically significant (p = 0.111). A similar pattern was evident for any victimisation [boys: OR 0.65 (95% CI 0.46 to 0.90); girls: OR 0.57 (95% CI 0.31 to 1.02)], but this difference was statistically significant (p = 0.018). Finally, although girls (OR 1.07, 95% CI 0.68 to 1.68) and boys (OR 1.27, 95% CI 0.85 to 1.90) had numerically different intervention impacts on the odds of forced sex, this difference was, again, not significant (p = 0.594). In the TSM group, neither boys (d = 0.01, 95% CI –0.13 to 0.16) nor girls (d = –0.04, 95% CI –0.29 to 0.21) experienced significant intervention impacts on the frequency of bullying; an interaction test did not suggest evidence of effect modification (p = 0.388). However, although both boys (OR 1.16, 95% CI 0.81 to 1.62) and girls (OR 1.75, 95% CI 0.97 to 3.14) experienced numerical, but not statistically significant, increases in any victimisation as a result of the intervention, there was evidence of a differential intervention effect favouring boys (p = 0.0001). As in the LSM group, neither boys (OR 0.89, 95% CI 0.57 to 1.38) nor girls (OR 1.06, 95% CI 0.65 to 1.74) experienced any intervention impacts on forced sex; a test of interaction did not find evidence of a differential intervention effect (p = 0.517).

Socioeconomic position

Only one trial, Learning Together,166,168 considered moderation by socioeconomic position for violence perpetration. In the main trial report,166 GBS scores at 36 months did not suggest a difference (p = 0.8944) in intervention effect between students with low levels of family affluence (MD –0.02, 95% CI –0.07 to 0.03) and students with high levels of family affluence (MD–0.03, 95% CI –0.06 to 0.01). Subsequent analyses168 did not find that family affluence moderated cyberbullying victimisation at 24 or 36 months.

Baseline risk

Three RCTs presented moderation or subgroup analyses by baseline risk for violence victimisation outcomes. 166,172,186

In Cooperative Learning,186 baseline risk was operationalised as student engagement with school. At about 5.5 months post baseline, a significant interaction between intervention condition and student engagement was found for bullying victimisation (β = 0.21, SE 0.09; p < 0.05), suggesting that the intervention was more effective at reducing victimisation among those with low baseline levels of school engagement.

In the main report of the GST trial,172 schools with a high prevalence of physical violence at baseline (OR 0.39, 95% CI 0.21 to 0.75) were not different from schools with a low prevalence of physical violence at baseline (OR 0.41, 95% CI 0.22 to 0.78) on the primary outcome of physical violence from school staff in the previous week (p = 0.9192).

In the main report of Learning Together,166 baseline risk was operationalised as experience of bullying victimisation, measured by the GBS, and aggression, measured by ESYTC scale; violence victimisation was also measured by the GBS. Students with high levels of baseline victimisation benefited from the intervention (MD –0.41, 95% CI –0.45 to –0.36) more than students with low levels of baseline victimisation (MD 0.15, 95% CI 0.12 to 0.18), for whom there was some evidence of an intervention-generated increase in victimisation. This difference was significant (p < 0.0001). Similarly, students with high levels of baseline aggression experienced greater declines in bullying victimisation as a result of the intervention (MD –0.06, 95% CI –0.10 to –0.02) than students with low levels of baseline aggression (MD 0.01, 95% CI –0.03 to 0.05), with a significant interaction effect (p = 0.0024).

Age

Only the report of the CDP163 reported subgroup analyses of intervention impacts on observed violence by sex. Drawing on teachers’ ratings of whether or not students insult others, get others into trouble, start fights or destroy others’ property, the analysis stratified children by grades 6, 7 and 8. No formal interaction tests were presented, and overall intervention impacts were not significant; grade-level means suggested benefits for students in grade 6 (M = 1.88 vs. M = 2.12 for intervention vs. control students, respectively), but harms for students in grade 7 (M = 1.93 vs. M = 1.68 for intervention vs. control students, respectively) and grade 8 (M = 1.94 vs. M = 1.90 for intervention vs. control students, respectively).

Sex

Only the Learning Together trial168 presented moderation of observed violence outcomes by sex. In this study, sex moderated observed aggression at 24, but not at 36, months post baseline. Girls (MD –0.15, 95% CI –0.26 to –0.04), but not boys (MD 0.01, 95% CI –0.11 to 0.13), reported lower levels of observed aggression; this interaction was significant (p = 0.02) (see Figure 48).

Ethnicity

Two studies presented moderation of observed violence outcomes by ethnicity. 85,173 In Positive Action Nevada,85 ethnicity (defined as percentage of African American students) did not moderate number of violent incidents, either in total per school or per 1000 students or when broken down to student-to-student violence, student-to-staff violence or possession of weapons; exact estimates of moderation or tests of moderation were not presented. In Positive Action Florida,173 schools with higher proportions of African American students experienced greater intervention effects in reductions of violent incidents per 100 students; in schools below the median for percentage of African American students, Positive Action was associated with a mean reduction in violent incidents of 0.9, whereas, in schools above the median, the mean reduction was 3.8. Although the interaction term was significant, an exact test was not presented.

Socioeconomic position

Two studies presented moderation of observed violence outcomes by socioeconomic position. 85,168 In the Learning Together RCT,168 family affluence moderated observed aggression at 24, but not at 36, months post baseline. At 24 months post baseline, intervention impacts on observed aggression exhibited a U-shaped relationship with family affluence; at both low levels (MD –0.18, 95% CI –0.62 to 0.25) and high levels (MD –0.14, 95% CI –0.25 to –0.03) of family affluence, the intervention reduced students’ reports of observed aggression, but this was not the case for students with a middle level of affluence (MD 0.04, 95% CI –0.09 to 0.18). The interaction test was significant (p = 0.03). In Positive Action Nevada,85 socioeconomic position, defined as school-level percentage of students with free/reduced-price lunch, did not moderate observed violence outcomes, although exact estimates of moderation or tests of moderation were not presented.

Age

Only one study163 presented subgroup estimates by age for substance use outcomes. An analysis of tobacco use, alcohol use, marijuana use or other illicit drug use in the previous 30 days was stratified by grade. No formal interaction tests were presented, and all intervention impacts were non-significant; scrutiny of grade-level estimates for each outcome revealed nearly identical prevalence between intervention and control group students. In both intervention and control groups, 6% of grade-6 students and 10% of grade-7 students reported tobacco use, whereas 12% of grade-8 intervention students and 13% of grade-8 control students reported tobacco use. Six per cent of grade-6 intervention students and 8% of grade-6 control students reported alcohol use, with similar absolute differences for grade-7 students (10% vs. 12% for the intervention and control groups, respectively) and grade-8 students (17% vs. 19% for the intervention and control groups, respectively). Marijuana use findings were inconsistent in direction between grade 6 (2% vs. 3% for the intervention and control groups, respectively), grade 7 (5% vs. 7% for the intervention and control groups, respectively) and grade 8 (9% vs. 8% for the intervention and control groups, respectively), but revealed minimal differences. Differences in illicit drug use were also minuscule for grade 6 (8% vs. 9% for the intervention and control groups, respectively), grade 7 (6% for both groups) and grade 8 (5% vs. 6% for the intervention and control groups, respectively).

Sex

Six RCTs reported moderation or subgroup analyses by sex for substance use outcomes. 64,87,117,166,168,176,181 Three studies87,117,166,168 demonstrated a pattern of intervention impacts favouring boys, whereas one study181 demonstrated a pattern of intervention impacts favouring girls. Two studies64,176 did not provide evidence of an intervention effect gradient by sex (see Figure 48).

In the RCT of the AAYP school/community intervention,87 estimates for growth in substance use over 4 school years were presented stratified by sex. A formal interaction test was not presented; however, boys in the school and community intervention group experienced decreased growth in substance use, compared with the control group, with a relative reduction by the end of grade 8 of 35% (p = 0.05). Girls in the intervention group experienced a non-significant decreased growth in substance use, compared with girls in the control group (48% vs. 52%, respectively; p = 0.86).

In the main report of DARE Plus,117 substance use outcomes [specifically, alcohol use in the previous year and previous month, ever drunk (i.e. intoxicated) and current smoker] were analysed via longitudinal models and stratified by sex, although no formal interaction tests were presented. Across outcomes, differences between groups were similar in direction and magnitude between boys and girls, although only effects for boys reached statistical significance. For alcohol use in the previous year, boys in DARE Plus had a marginally significant (one-tailed p = 0.04) decrease in rate of growth, compared with boys in the control group (β = 0.19 vs. β = 0.26, respectively). An intervention effect for girls was not reported on this outcome (β = 0.23 vs. β = 0.25 for the intervention and control groups, respectively; one-tailed p = 0.36). For alcohol use in the previous month, boys’ rate of growth was significantly lower (one-tailed p = 0.01) in the DARE Plus group (β = 0.08) than in the control group (β = 0.14); for girls, the difference was of similar magnitude and direction (β = 0.08 vs. β = 0.12 for the intervention and control groups, respectively), but not significant (one-tailed p = 0.15). The growth rate for the outcome of ever drunk was not different for boys across both groups (β = 0.11 vs. β = 0.15 for the intervention and control groups, respectively; one-tailed p = 0.07), nor for girls (β = 0.07 vs. β = 0.12 for the intervention and control groups, respectively; one-tailed p = 0.11). Finally, boys in the DARE Plus group experienced a decrease in growth of current smoking, compared with boys in the control group (β = 0.18 vs. β = 0.31 for the intervention and control groups, respectively; one-tailed p = 0.02); girls did not demonstrate an intervention effect on this outcome (β = 0.22 vs. β = 0.28 for the intervention and control groups, respectively; one-tailed p = 0.25).

Although the main report of Learning Together166 tested a range of substance use outcomes at 36 months post baseline, moderation analyses by sex were presented for outcomes of ever smoked, ever drank alcohol and illicit drug use (a three-category variable comprising ‘no’, ‘yes, offered but did not accept’ and ‘yes, offered and accepted’). There was clear evidence of moderation by sex for smoking and alcohol use, with greater intervention effects for boys. For the outcome of ever smoked, boys in the intervention group reported lower odds of smoking than boys in the control group (OR 0.33, 95% CI 0.22 to 0.50), whereas differences were not significant for girls (OR 0.87, 95% CI 0.60 to 1.25), with a significant interaction test (p < 0.0001). A similar pattern was in evidence for alcohol use [boys: OR 0.52 (95% CI 0.38 to 0.70); girls: OR 0.95 (95% CI 0.71 to 1.26)], again with a significant interaction test (p = 0.0002). Estimates of intervention impact were closer between boys and girls for the illicit drug use outcome [boys: OR 0.44 (95% CI 0.29 to 0.68); girls: OR 0.57 (95% CI 0.38 to 0.86)]. Although the intervention was statistically effective for both groups on this outcome, there was no evidence of interaction (p = 0.2350). A subsequent analysis168 examined e-cigarette use and found that Learning Together was more effective for boys than for girls at both 24 months [boys: OR 0.45 (95% CI 0.30 to 0.67) vs. girls: OR 0.80 (95% CI 0.83 to 1.55); p = 0.014] and 36 months [boys: OR 0.35 (95% CI 0.23 to 0.53) vs. girls: OR 0.94 (95% CI 0.63 to 1.40); p < 0.001].

The Hawaii trial of Positive Action64 presented substance use outcomes for boys and girls both as individual behaviours and as a summary score; however, only the summary score included a formal test of interaction. Individual substance use behaviours did not show a clear pattern of greater effect for boys or girls; smoking a cigarette (boys: OR 0.66; girls: OR 0.38) and using an illegal drug (boys: OR 0.34; girls: OR 0.15) both reflected greater intervention benefits for girls, but the opposite was the case for getting drunk on alcohol (boys: OR 0.24; girls: OR 0.40) and getting high on drugs (boys: OR 0.18; girls: OR 0.25), with nearly identical effects for drinking any alcohol (boys: OR 0.48; girls: OR 0.47). The overall interaction test on the sum score of substance use behaviours suggested a non-significant differential intervention impact in reducing substance use among boys, compared with girls (interaction IRR 1.07, 95% CI 0.65 to 1.80; p = 0.402). Teacher reports of smoking (boys: OR 0.42; girls: OR 0.78), drinking (boys: OR 0.66; girls: OR 1.16) or illicit drug use (boys: OR 0.21; girls: OR 0.42) for individual students did, however, suggest a pattern of greater intervention effects for boys, albeit with no formal interaction tests.

In the SEHER trial,181 substance use outcomes were evaluated at 8 months post baseline, and included smoking tobacco, chewing tobacco, drinking alcohol and other substance use. In the LSM arm, compared with the control arm, effects were consistently greater for girls than for boys. Girls demonstrated a greater intervention-generated decrease in smoking tobacco (OR 0.56, 95% CI 0.25 to 1.23) than boys (OR 1.16, 95% CI 0.69 to 1.92), with a significant interaction test (p < 0.0001). Effects were numerically similar for chewing tobacco [girls: OR 0.54 (95% CI 0.29 to 0.98); boys: OR 1.12 (95% CI 0.65 to 1.91)], with a significant interaction test (p < 0.0001); for drinking alcohol [girls: OR 0.53 (95% CI 0.27 to 1.03) vs. boys: OR 1.05 (95% CI 0.62 to 1.78); p = 0.020]; and for other substance use [girls: OR 0.64 (95% CI 0.26 to 1.57) vs. boys: OR 0.99 (95% CI 0.72 to 1.34); p = 0.002]. Differences in intervention impact between boys and girls were all non-significant when comparing the TSM group with the control group. For alcohol and tobacco outcomes, effects were numerically greater for girls than for boys, but tended to be closer in magnitude than in the LSM arm: smoking tobacco – [girls: OR 0.45 (95% CI 0.19 to 1.05) vs. boys: OR 0.74 (95% CI 0.51 to 1.07) (p = 0.362)], chewing tobacco – [girls: OR 0.49 (95% CI 0.26 to 0.93) vs. boys: OR 0.60 (95% CI 0.33 to 1.07) (p = 0.341)] and drinking alcohol – [girls: OR 0.45 (95% CI 0.21 to 0.93) vs. boys: OR 0.80 (95% CI 0.45 to 1.43) (p = 0.157)]. However, in the TSM arm, intervention impacts for other substance use were greater for boys (OR 0.74, 95% CI 0.51 to 1.07) than for girls (OR 0.91, 95% CI 0.35 to 2.31), albeit with a non-significant interaction test (p = 0.501).

Finally, in the whole-of-school intervention,176 moderation was tested by gender for a range of substance use outcomes measured at 35 months post baseline. There was no evidence of moderation (p > 0.05 in every case) by gender for any substance use outcomes, including recent tobacco use [boys: OR 1.37 (95% CI 0.77 to 2.42); girls: OR 1.38 (95% CI 0.78 to 2.43)], amount of tobacco use [boys: MD –4.56 (95% CI –19.35 to 10.23); girls: MD –0.33 (95% CI –13.69 to 13.03)], recent alcohol use [boys: OR 1.09 (95% CI 0.72 to 1.66); girls: OR 1.12 (95% CI 0.72 to 1.73)], recent risky alcohol use [boys: OR 0.80 (95% CI 0.55 to 1.16); girls: OR 1.21 (95% CI 0.82 to 1.79)], amount of alcohol consumed [boys: MD –1.40 (95% CI –4.29 to 1.49); girls: MD 0 (95% CI –3.01 to 3.00)], any marijuana use [boys: OR 1.04 (95% CI 0.65 to 1.67); girls: OR 1.23 (95% CI 0.74 to 2.04)] or any other substance use [boys: OR 0.96 (95% CI 0.51 to 1.82); girls: OR 1.77 (95% CI 0.80 to 3.92)].

Socioeconomic position

Two studies166,168,176 presented moderation analyses by socioeconomic position. Neither analysis suggested a pattern of differential effects.

There was little evidence of moderation by family affluence in Learning Together,166 with no significant evidence of interaction for outcomes of ever smoked, ever drank alcohol and illicit drug use. For the outcome of ever smoked, there was a numerically greater intervention benefit for students of lower socioeconomic status at baseline (OR 0.51, 95% CI 0.32 to 0.81) than for students of high economic status at baseline (OR 0.66, 95% CI 0.46 to 0.95), but no significant interaction effect (p = 0.3003). For the outcome of ever drank alcohol, the numerical pattern was similar [low socioeconomic status: OR 0.59 (95% CI 0.41 to 0.85); high socioeconomic status: OR 0.81 (95% CI 0.61 to 1.08)], with a marginally significant interaction test (p = 0.0869). Intervention effect estimates were nearly identical for illicit drug use [low socioeconomic status: OR 0.50 (95% CI 0.31 to 0.81); high socioeconomic status: OR 0.54 (95% CI 0.36 to 0.80)], with no interaction effect (p = 0.7599). A subsequent analysis focusing on e-cigarette use168 did not find evidence of moderation by family affluence at 24 or 36 months post baseline. In the whole-of-school intervention,176 moderation was tested by family affluence for a range of substance use outcomes measured at 35 months post baseline. Only intervention impacts on amount of tobacco was moderated by family affluence (p < 0.01), with marginally significant intervention impacts on students with a low socioeconomic position (MD –12.89, 95% CI –26.00 to 0.23), but a non-significant intervention-generated increase among students of high socioeconomic position (MD 16.36, 95% CI –1.03 to 33.76). There was no evidence of moderation (p > 0.05 in every case) by family affluence for any other substance use outcomes, including recent tobacco use [low socioeconomic status: OR 1.40 (95% CI 0.79 to 2.48); high socioeconomic status: OR 1.35 (95% CI 0.65 to 2.81), recent alcohol use (low socioeconomic status: OR 1.06 (95% CI 0.70 to 1.61); high socioeconomic status: OR 1.21 (95% CI 0.71 to 2.07)], recent risky alcohol use [low socioeconomic status: OR 1.06 (95% CI 0.72 to 1.55); high socioeconomic status: OR 0.84 (95% CI 0.52 to 1.35)], amount of alcohol consumed [low socioeconomic status: MD –0.91 (95% CI –3.65 to 1.84); high socioeconomic status: MD –0.59 (95% CI –4.20 to 3.02)], any marijuana use [low socioeconomic status: OR 1.30 (95% CI 0.79 to 2.15); high socioeconomic status: OR 0.85 (95% CI 0.46 to 1.56)] or any other substance use [low socioeconomic status: OR 0.47 (95% CI 0.72 to 3.01); high socioeconomic status: OR 0.82 (95% CI 0.34 to 1.94)].

Baseline risk

Randomised controlled trials of the Gatehouse Project, Learning Together and whole-of-school interventions68,166,176 present moderation analyses for substance use by baseline risk. The baseline risk variables considered included school engagement, baseline substance use,176 and bullying victimisation and aggression. 166

In the Gatehouse Project,68 incidence and prevalence of cannabis use were compared over strata defined by school engagement and baseline tobacco use. By year 10 (after 3 school years of the intervention), there was no evidence of an interaction effect between intervention group and baseline tobacco smoking for either prevalence (p = 0.22) or incidence (p = 0.25) of any cannabis use. Weekly cannabis use was also considered as an outcome. Although there was no evidence of moderation by baseline tobacco smoking for prevalence of weekly cannabis use (p = 0.10), baseline tobacco smoking did moderate intervention effects on incident weekly cannabis use (p = 0.04), with evidence of a significant intervention effect for baseline non-smokers (OR 0.50, 95% CI 0.26 to 0.98). School engagement did not yield a significant moderation test for intervention effects for prevalence of any cannabis use (p = 0.84), incidence of any cannabis use (p = 0.55), prevalence of weekly cannabis use (p = 0.97) or incidence of weekly cannabis use (p = 0.84).

In Learning Together,166 moderation of intervention effects for outcomes of ever smoked, ever drank alcohol and illicit drug use at 36 months post baseline was tested using baseline risk stratifiers of bullying victimisation and aggression. Baseline levels of bullying victimisation (i.e. low or high) did not moderate intervention effects for smoking [low: OR 0.66 (95% CI 0.46 to 0.96) vs. high: OR 0.52 (95% CI 0.33 to 0.80); p = 0.2887]; alcohol consumption, although an interaction test was marginally significant [low: OR 0.81 (95% CI 0.61 to 1.07) vs. high: OR 0.57 (95% CI 0.41 to 0.81); p = 0.0541]; or illicit drug use [low: OR 0.46 (95% CI 0.31 to 0.68) vs. high: OR 0.58 (95% CI 0.37 to 0.93); p = 0.2895]. However, baseline levels of aggression (i.e. low or high) did moderate intervention impacts on smoking [low: OR 0.93 (95% CI 0.58 to 1.49) vs. high: OR 0.46 (95% CI 0.32 to 0.68); p = 0.0053] and alcohol use [low: OR 1.08 (95% CI 0.78 to 1.49) vs. high: OR 0.56 (95% CI 0.41 to 0.74); p = 0.0009], with greater intervention effects for students with a high level of baseline aggression. However, baseline levels of aggression did not moderate intervention impacts on illicit drug use [low: OR 0.62 (95% CI 0.40 to 0.96) vs. high: OR 0.50 (95% CI 0.34 to 0.74); p = 0.3137].

Finally, in the whole-of-school intervention,176 moderation of each substance use outcome was tested by whether or not students were baseline users of the relevant substance. There was no evidence of moderation (p > 0.05 in every case) for any substance use outcomes by baseline use of that substance, including recent tobacco use [non-user: OR 1.39 (95% CI 0.83 to 2.31); user: OR 1.34 (95% CI 0.60 to 2.97)], amount of tobacco use [non-user: MD –5.16 (95% CI –16.55 to 6.23); user: MD 6.83 (95% CI –11.76 to 25.43)], recent alcohol use [non-user: OR 1.22 (95% CI 0.76 to 1.95); user: OR 0.94 (95% CI 0.63 to 1.42)], recent risky alcohol use [non-user: OR 0.99 (95% CI 0.69 to 1.41); user: OR 0.79 (95% CI 0.52 to 1.21)], amount of alcohol consumed [non-user: MD –1.05 (95% CI –3.76 to 1.67); user: MD –0.95 (95% CI –4.10 to 2.20)] or any marijuana use [non-user: OR 1.37 (95% CI 0.13 to 14.23); user: OR 1.09 (95% CI 0.69 to 1.72)]. No moderation analysis was possible for the outcome of any other substance use.

Age

Only one study, the CDP,163 presented subgroup estimates by age for academic attainment outcomes. The analysis of grade-point average and of achievement test score was stratified by grade. No formal interaction tests were presented, and all intervention impacts were non-significant. There was no clear pattern of magnitude or direction of effect by grade for either outcome. For example, for grade-point average, grade-6 students in the intervention group had a higher score than those in the control group (M = 2.52 vs. M = 2.39, respectively), but this effect was reversed for students in grade 7 (M = 2.34 vs. M = 2.39 for the intervention and control groups, respectively), and then reversed again for students in grade 8 (M = 2.50 vs. M = 2.43 for the intervention and control groups, respectively), with no effect reaching statistical significance.

Sex

Only the Chicago trial of Positive Action61 presented moderation of academic attainment outcomes by sex. Findings for moderation by sex of intervention impacts on student self-reported grades were not presented, but probably did not suggest differential impacts, given null effects overall. Impacts on teacher-rated academic performance were greater for boys than for girls, although the amount of this was not quantified. End-point effect sizes comparing intervention impacts for boys and girls on standardised test performance suggested a greater increase for boys (d = 0.33) than for girls (d = 0.11) on reading, but a greater impact for girls (d = 0.41) than for boys (d = 0.31) on mathematics. However, these differences were unlikely to be significant (see Figure 48).

Ethnicity

Moderation of academic attainment outcomes by ethnicity was tested in the non-randomised evaluation of Positive Action in Nevada and Hawaii. 85 Subgroup analyses were presented in the RCT of Positive Action in Chicago. 61 Findings were inconsistent, but generally appeared to be null.

In the study of Positive Action Nevada,85 ethnicity (defined as percentage of African American students) did not appear to moderate academic achievement on standardised tests, either in terms of mathematics; reading; language; science; or combined mathematics, reading and language scores. Similarly, in the Nevada and Hawaii analysis,85 ethnicity (defined as percentage of Japanese or Chinese students) did not appear to moderate academic achievement in standardised tests of mathematics or reading, or a combined measure of both. In the Chicago trial of Positive Action,61 end-point effect sizes comparing intervention impacts for subgroups were presented for standardised tests of reading and mathematics. African American students had larger effect sizes than the general population for both reading (d = 0.50 vs. d = 0.22, respectively) and mathematics (d = 0.55 vs. d = 0.38, respectively), but moderation tests, which were not formally presented, were unlikely to be significant, given a pattern of null effects for longitudinal models.

Socioeconomic position

Moderation of academic attainment outcomes by socioeconomic position was tested in the non-randomised evaluations of Positive Action in Nevada and Hawaii85 and in Florida,173 and subgroup analyses were presented in the Chicago trial of Positive Action. 61 Although reporting was poor, no studies suggested effectiveness gradients by socioeconomic position.

Although reporting was unclear in the non-randomised evaluations of Positive Action in Nevada and Hawaii,85 it appeared that socioeconomic position (defined as school-level percentage of students receiving free or reduced-price lunch) did not moderate intervention effects on academic attainment as measured by standardised test performance. In the matched controls analysis of schools in the Florida evaluation of Positive Action,173 the percentage of students receiving free/reduced-price lunch did not moderate intervention impacts on standardised tests of academic achievement, although the exact outcome used was unclear. In the Chicago trial of Positive Action,61 end-point effect sizes comparing intervention impacts for subgroups were presented for standardised tests of reading and mathematics. Students receiving free or reduced-price lunch were not different from the general population in terms of effect sizes for reading (d = 0.23 vs. d = 0.22, respectively) or mathematics (d = 0.42 vs. d = 0.38, respectively).

Violence perpetration

All three trials presented mediation analyses for violence perpetration. Findings were inconsistent by measure of school commitment. Findings were inconsistent across studies.

In Learning Together,165 analyses tested whether or not student view of school climate, measured at 24 months using the Beyond Blue School Climate Questionnaire, mediated impacts on aggression perpetration (as measured by ESYTC scale) and bullying perpetration (modified aggression subscale). Although student view of school climate as a whole was linked to both aggression perpetration (adjusted MD –2.57, 95% CI –2.96 to –2.18) and bullying perpetration (adjusted MD –1.53, 95% CI –1.74 to –1.33), including student view of school climate in regression models for either aggression perpetration or bullying perpetration did not appear to attenuate intervention impacts, suggesting that student view of school climate did not mediate violence perpetration outcomes. In the SEHER trial,184 school climate, measured at 8 months post baseline, was tested for mediation of violence perpetration outcomes at 17 months post baseline. Neither sense of belonging nor commitment to school was linked to the outcome. However, student report of school climate as a whole significantly mediated reductions in violence perpetration (indirect effect β = –0.081, 95% CI –0.100 to –0.034), reflecting 15.43% of the total intervention effect on this outcome. Finally, Positive Action Hawaii124 included a test of whether or not school commitment mediated violence perpetration, measured as a count of violence-related behaviours. A full mediation model demonstrated that student reports of academic behaviours accounted for a 42.3% reduction in violent behaviours (p < 0.01), with teacher-rated academic behaviours accounting for a 16.0% reduction in violent behaviours (p < 0.05).

Violence victimisation

Two trials, Learning Together165 and SEHER,184 presented mediation analyses for violence victimisation. In Learning Together,165 analyses of mediation on bullying victimisation at 36 months post baseline did not demonstrate attenuation of intervention impacts when student view of school climate was included in regression models, suggesting that student view of school climate did not mediate violence victimisation outcomes. In the SEHER trial,184 sense of belonging (indirect effect β = –0.033, 95% CI –0.057 to –0.007), but not commitment to school, mediated intervention impacts on bullying victimisation. An overall measure of school climate had a significant indirect effect on victimisation, accounting for 14.87% of intervention impacts on victimisation.

Substance use

Two trials, Learning Together165 and Positive Action Hawaii,124 presented mediation analyses for substance use. In Learning Together,165 analyses of mediation on substance use outcomes (ever smoked, ever drank alcohol, ever been really drunk, ever been offered illicit drugs) at 36 months post baseline did not demonstrate attenuation of intervention impacts when student view of school climate was included in regression models, suggesting that student view of school climate did not mediate substance use outcomes. In Positive Action Hawaii,124 student-rated academic behaviours completely mediated the impact of Positive Action on count of substance use behaviours, accounting for a 44.6% reduction in substance use and generating a statistically non-significant direct effect of the intervention. Academic behaviours rated by teachers generated only partial mediation, but still accounted for a 14.9% reduction in substance use behaviours.