Training teachers in classroom management to improve mental health in primary school children: the STARS cluster RCT

Article history

The research reported in this issue of the journal was funded by the PHR programme as project number 10/3006/07. The contractual start date was in June 2012. The final report began editorial review in March 2018 and was accepted for publication in September 2018. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PHR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Stuart Logan was a member of the Health Technology Assessment (HTA) Commissioning Board 2003–10, the HTA Medicines for Children Themed Call 2005–6 and the Rapid Trials and Add on Studies Board 2012.

Copyright statement

© Queen’s Printer and Controller of HMSO 2019. This work was produced by Ford et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2019 Queen’s Printer and Controller of HMSO

Disruption in school

Disruption in the classroom can undermine the quality of teaching and interrupt the learning of all children in a class. 1 Conduct disorder (or behaviour that consistently violates social norms to the extent that a child’s ability to function is impaired) is the most common childhood mental health problem, with a prevalence of 5–6% in the UK and high levels of comorbidity. 2,3 Furthermore, levels of disruptive behaviours are normally distributed throughout the population. 4 This means that in each classroom in the UK there is likely to be at least one child with a diagnosable conduct disorder, as well as several others who are disruptive to their classmates and teachers at lower levels. 5 Parent training is the evidence-based treatment for childhood behaviour problems6 but this has limited impact on school-based problems. 7 A recent systematic review of teacher-led interventions for internalising and externalising symptoms reported weak evidence for the effectiveness of interventions on internalising problems (Cohen’s d = 0.13) and no evidence for programmes on externalising problems. 8

Teachers report that disruptive behaviour and the task of managing the classroom can lead to high levels of stress and burnout. 9–12 Teachers themselves note the lack of training that they receive in the area of classroom management. 13 Improving the skills of one teacher to manage their classroom effectively would have an impact not only on the children they currently teach but also on children they will teach in the future: assuming one primary school teacher has a new class of 30 children every 2–3 years, within 10 years they will teach > 100 individuals.

A universal intervention that trains teachers has the potential to reach many thousands of children. Interventions that promote socioemotional competencies will affect children at high risk of later mental health problems, as well as their peers who may not have early indicators of risk but who still make up a substantial proportion of those with later mental health difficulties. 14 Promoting resilience at an early stage in life could lead to a population-level reduction in the burden and cost associated with mental ill health. In addition, poor mental health is highly associated with lower levels of academic attainment,15–17 and recent work has demonstrated that improving a child’s mental health will lead to subsequent improvements in their levels of attainment. 18 Therefore, an effective intervention that can be applied and delivered in an ordinary school setting, with no cost other than to train the teachers delivering it, would incur minimal costs for a wide-reaching population health benefit.

Universal interventions in the school setting: the Incredible Years^® Teacher Classroom Management programme

There are a variety of universal interventions in existence that are delivered in school settings in order to promote child socioemotional competencies and minimise disruptive behaviour. Those that are led by teachers, and, therefore, may benefit many children rather than targeting individual children who are ‘high risk’, are more limited. A systematic review of teacher-led interventions that target children’s social and emotional behaviour found that only two existing programmes had been evaluated in a controlled design (randomised and non-randomised or pre–post quasi-experimental designs): the Incredible Years^® (IY) Teacher Classroom Management (TCM) programme,19 and the Good Behaviour Game. 20,21 Of these two programmes, the systematic review found that the evidence for TCM was the most robust. 21 Importantly, this programme aims not only to improve the socioemotional skills and behaviour of children with externalising disorders but also to promote socioemotional skills in all children, with the aim of preventing poor outcomes in the future. 19

Description of Teacher Classroom Management training

Like the IY parent and child programmes, TCM training was initially developed for individual children with conduct disorders but has been extended to children with risk factors that increase their likelihood of going on to develop a psychiatric disorder or to have other poor social outcomes, such as delinquency or substance use. 22 It is believed that, by helping teachers to support these children, TCM will reduce disruption to the rest of the class, leading to a general improvement in all of their pupils’ socioemotional skills. The focus of the TCM training is on collaborative learning, discussions of teachers’ own experiences and group work to find solutions to problems encountered in the classroom.

The TCM training has four explicit goals:

1. to enhance teacher behaviour-management skills and improve teacher–pupil relationships

2. to help teachers to develop effective individual and group behaviour plans in order to enable proactive (as opposed to reactive) classroom management

3. to encourage teachers to adopt and promote social and emotional regulation skills

4. to encourage teachers to strengthen positive teacher–parent relationships (the promotion of positive relationships between parents, children and teachers is a central tenet of the IY series). 23

Research to date suggests that, although teachers already use a lot of these techniques, TCM training can increase their skills and confidence in using them sufficiently to produce significant improvements in the behaviour of their pupils. TCM uses a range of methods to deliver manualised training designed to improve classroom management. TCM draws on cognitive social learning theory, particularly Patterson’s theories24 about how coercive cycles of interaction between adults and children reinforce unwanted behaviour patterns, Bandura’s ideas25 about the importance of modelling and self-efficacy, and Piaget and Inhelder’s developmental interactive learning methods. 26 In addition, it incorporates strategies for challenging angry, negative and depressive internal dialogue in adults while interacting with children that are drawn from cognitive behavioural approaches. 27

The delivery of the TCM teaching objectives follows these theoretical perspectives and includes problem-solving, role-play, modelling, goal setting, reflective learning, group discussion and support. Cognitive and emotional self-regulation training is also included in the course. 23

The manualised version of TCM is intended to be delivered in a collaborative style to groups of teachers by a trained group leader over 6 full days, with time between each session for teachers to practise the new strategies they have learnt. 23 Table 1 outlines the key concepts that are covered in each of the six TCM workshops.

We have proposed a number of mechanisms of change in the logic model shown in Figure 1. We hypothesise that TCM will produce changes in the teachers’ behaviour in the classroom that will lead to positive changes in the children within the classroom. We anticipate that any changes that the teacher makes will have an impact both on individual children and on their class as a whole. It is likely that a reinforcement loop will be in operation, exemplifying positive changes between all three groups. It is important to consider the wider context and what impact this may have on TCM’s mechanisms of change. Certainly, the identified external factors may support or damage this proposed theory of change, and these factors are ones that are explored as part of the process evaluation work.

FIGURE 1.

Teacher Classroom Management training logic model highlighting the proposed mechanisms of change. ↑, increased; ↓, reduced.

Published studies using TCM vary in terms of whether they adhere strictly to the advised training and implementation or adapt the TCM training for individual contexts, as well as in terms of whether TCM is applied as a stand-alone intervention or in conjunction with other IY components or adaptations.

Existing research on the Teacher Classroom Management programme

At the time of writing, 21 independent data sets had been identified that report on the use of the TCM training, and two reviews had been identified that each aimed to collate the evidence around TCM. 28,29

Three initial randomised controlled trials (RCTs) conducted by the IY developers tested the effects of the TCM training in conjunction with other IY components in high-risk samples. The studies reported that children in the intervention arms had significantly fewer teacher-reported symptoms of hyperactivity and antisocial behaviour,22 more social competencies22,30 and better emotion regulation30 and were more co-operative with their teachers and less aggressive with peers. 31 Teachers who received the TCM training were observed to use more positive and fewer harsh and critical techniques,22 and more praise, as well as to be more confident, consistent and nurturing. 31 There was tentative evidence that classrooms with higher levels of conduct problems benefited most from the intervention. 30 As these initial studies evaluated TCM with the parallel parent programmes or mentoring support, it is impossible to know which IY components were responsible for changes in child and teacher behaviour.

Study design

The STARS trial was a two-arm, pragmatic, parallel-group, superiority, cluster RCT designed to evaluate whether TCM training (delivered at class level) improves the mental health of individual children. Schools were randomly allocated to the intervention arm (TCM training) or the control arm [teaching as usual (TAU)]. The trial included a parallel economic evaluation to examine the cost-effectiveness of the TCM training and a mixed-method process evaluation that used qualitative methods to assess the acceptability of the TCM training; these are reported on in Chapters 4 and 6, respectively.

Setting, participants and recruitment

We recruited 80 primary schools across the south-west of England in Devon, Torbay and Plymouth in three separate cohorts: September 2012 (cohort 1), September 2013 (cohort 2) and September 2014 (cohort 3). One class (teacher and all pupils) was selected by the headteacher from each school independently of the research team.

Inclusion and exclusion criteria

Schools were considered for inclusion if they had a single-year class with ≥ 15 children aged between 4 and 9 years who were taught by a teacher who held classroom responsibility for at least 4 days per week. Schools were excluded if they primarily taught pupils with special educational needs (SEN), lacked a substantive headteacher or had been judged as failing at their last Office for Standards in Education, Children’s Services and Skills (Ofsted) inspection.

All children in the selected classes were eligible for inclusion provided that the class teacher judged that they and their parents had sufficient English-language comprehension to understand recruitment information and complete outcome measures.

Randomisation and concealment

Randomisation of schools, using computer-generated random numbers, was carried out by an independent researcher based at the University of Exeter who was masked to the identity of the schools to ensure allocation concealment. The allocation was passed on to the trial manager, who then informed the schools. Randomisation was completed after baseline data collection to avoid recruitment and response bias. 46 Allocation was completed separately for each cohort, with all schools in a given cohort allocated en bloc. An equal number of schools were allocated to each arm overall, but unequal allocation ratios were used for cohorts 1 and 3 to ensure that there was an adequate number of intervention teachers to fill each TCM training group, with the ratio of intervention to TAU schools being 10 : 5 in cohort 1, 15 : 15 in cohort 2 and 15 : 20 in cohort 3.

Allocation was balanced on the following school factors: urban versus rural/semi-rural area, key stage (KS) 1 (Reception to Year 2) versus KS 2 (Years 3 or 4) and deprivation (whether the percentage of children eligible for free school meals was > 19%, the UK national average in 201247). Because there were relatively few clusters within each cohort, there was a high chance that random allocation using standard stratification or minimisation methods would not have yielded trial arms that were similar on the balancing factors. To overcome this, we used the approach of randomly selecting an allocation sequence for each cohort from the 5% with the least imbalance out of one million randomly generated potential allocations (permutations). Imbalance was quantified by the sum of the mean differences on the three balancing factors between trial arms weighted by the inverse of the variance of those factors. 48

We were unable to mask the schools and teachers because the school needed to release the class teacher to attend the training. Children and parents were not informed about whether or not the teacher attended training. Baseline measures were completed before randomisation. Even following randomisation, parents and children were unlikely to be aware of whether or not their child’s teacher had completed TCM training. The follow-up measures were questionnaires that were completed independently of the researchers (with the exception of the service-related interviews, which were completed by a subsample of parents, and the child measures, as younger children might require support) and thus difficult for the core team of researchers to influence. In addition, the teacher-completed follow-up measures in the second and third years of each school’s participation in the study (18 and 30 months post baseline) were completed by a teacher who did not access the intervention, although they were likely to have known whether their colleague who taught the class in the first academic year of the study did or did not.

Data collection

All baseline (T0) measures were collected before the end of the first academic half term of the schools’ involvement, usually in October. Follow-up measures were then completed 9 [June, time point 1 (T1)], 18 [February, time point 2 (T2)] and 30 [February, time point 3 (T3)] months later. The 9-month time point was chosen to capture any initial impact the training may have had. Longer-term follow-up was important to see if potential impacts were sustained, reduced or potentially accelerated and the 18- and 30-month time points were chosen as they were mid-way through the following two academic years. Although additional time points during the first year would have been beneficial to track potential change, including any mediators of this change, we were mindful of the need to reduce the response burden on teachers, parents and children, and, therefore, we chose fewer time points that were optimally placed to capture change. Baseline and 9-month assessments took place during the first academic year of participation, before and after the intervention, respectively, so were completed by the same teacher at both these time points. The 18-month and 30-month assessments occurred during the children’s second and third academic year of the trial and were therefore completed by different teachers (Figure 2). It was not possible to ask the original teacher to complete the 18- and 30-month assessments because, in order to complete the measures, teachers must spend a large amount of time with the child to accurately assess their current development. Teacher measures were completed on an online database built for the trial (see Report Supplementary Material 1). To enable a supply teacher to supervise the teacher’s class, schools received £80 for each time point at which teachers completed the outcomes (£320 in total) and £160 for each training day attended (£960 in total). Teachers also personally received a £10 gift voucher after outcome completion at each wave. Parent questionnaires were sent home with participating children. Parents received reminders via the school office and, where possible, second questionnaires were posted directly to the home. Parents received a £5 gift voucher for every completed questionnaire (£20 in total). Child-reported outcome data were collected during school time by researchers as a classroom activity for children aged ≥ 7 years, or individually for younger children. School staff were present but instructed not to assist the children.

FIGURE 2.

Schematic detailing the timing of outcome measures. APP, assessment of pupil progress; CA-SUS, Child and Adolescent Service Use Schedule; HIFAMS, How I Feel About My School; PBQ, Pupil Behaviour Questionnaire.

Outcome measures

Sample size

As previously reported in our procotol,57 40 schools (clusters) were randomised to each of the intervention and TAU arms, using one class (teacher and pupils) from each school. Assuming that each class contains 30 pupils and that the recruitment rate is 70% (achieved among parents in the Helping Children Achieve trial58), we anticipated that 21 (i.e. 30 × 0.7) children from each class of 30 and a total of 840 (i.e. 21 × 40) children in each trial arm would participate in the study. Assuming 10% attrition for the children, we expected 19 of them to be followed up at T3 in each class and a total of 760 (i.e. 19 × 40) children to be followed up at T3 in each trial arm. As clusters were randomised, the sample size was calculated taking account of the correlation between pupils’ responses within clusters (or equivalently the variation between clusters). The variance inflation factor (VIF) (design effect) is given by VIF = 1 + [(n-1) × ICC], presented in Donner and Klar,59 where n is the number of pupils providing outcome data at follow-up in each school and ICC is the intracluster (intraschool) correlation coefficient for the primary outcome. Nineteen children in each school were expected to provide follow-up data at T3 and the ICC for the SDQ-TD score (the primary outcome) was assumed to be 0.15 based on analysis of data from Sayal et al. 60 The assumed ICC value takes account of both the inherent variability across schools and the additional variability resulting from the fact that only one classroom was included in the study from each school. The VIF was 3.7 {i.e. 1 + [(19 – 1) × 0.15]} and the target sample size therefore has the same effective sample size as an individually randomised trial with 205 (= 760/3.7) participating pupils at follow-up in each arm, providing 85% power at the 5% level of significance to detect a difference between trial arms of 0.3 standard deviation (SD) units (Cohen’s d = 0.3), or a difference of 1.8 points on the raw SDQ scale (the SD of the teacher-reported SDQ has been estimated to be 5.9 for 4801 UK children aged 5–10 years). 61 This effect would reduce the percentage of children classified in the borderline/abnormal range from 19.7% to 13.7%,61 where borderline/abnormal is defined as those scoring 12 or more out of 40 on the teacher-reported SDQ-TD score. Data from Goodman and Goodman49 suggest that the odds of psychiatric disorder decrease by 33% for each two-point decrease on the teacher SDQ and by 40% for each two-point decrease on the parent SDQ.

Statistical analysis

All analyses, performed using Stata^® version 14.2 (StataCorp LP, College Station, TX, USA) software, were pre-specified in a statistical analysis plan that was reviewed by the independent Data Monitoring Committee (DMC) and the Trial Steering Committee (TSC).

Baseline characteristics of the schools, teachers and children were summarised for each trial arm, reporting means and SDs [or medians and interquartile ranges (IQRs)] for quantitative variables, and numbers and percentages for categorical variables. The characteristics of participating schools were compared with those reported in the 2012 school census for England. 47

The trial outcomes at follow-up were compared using the intention-to-treat principle; children were analysed strictly according to the trial arm to which their school was randomised. The main findings presented are based on analyses of complete cases. In addition, we carried out sensitivity analyses based on 50 multiply imputed data sets using the chained equations approach. 62

Quantitative outcomes were compared between trial arms using random-effects linear regression models, and binary outcomes were compared using marginal logistic regression models using generalised estimating equations with information sandwich (‘robust’) estimates of standard error assuming an exchangeable correlation structure. These methods allow for the correlation of children’s outcome scores within schools. The primary analyses were those in which potential confounders were adjusted for, specified a priori in the analysis plan as the following: the three school-/class-level factors used to balance the randomisation, cohort, child gender, baseline score of the outcome, Index of Multiple Deprivation (IMD) score based on the child’s address, number of children living in their household and whether or not the child’s household was rented. The last three of these nine prognostic factors had a large number of missing data because they were parent reported. Adjusting for these would have resulted in the loss of one-quarter of the sample in the complete-case analyses (CC1). On this basis, and after discussion with our DMC (5 June 2017), we agreed the primary analysis as the CC1 adjusted for only the six a priori prognostic factors that were not parent reported (CC1). For completeness, we also report the findings from the complete-case sensitivity analysis (CC2) and the analysis of imputed data with all nine prognostic factors included [multiple imputation (MI)].

For all outcomes, tests of interaction were used to assess whether or not there was evidence that the effect of the TCM intervention differed across the three follow-up time points. When the interaction effect is statistically significant at the 5% level, we report the effect at each wave; otherwise, we report an estimate of the average intervention effect across the three follow-up waves.

In an ancillary analysis we used the two-stage least squares instrumental variable method63 to calculate the complier average causal effect (CACE) estimate of the intervention effect on the primary outcome teacher-reported SDQ-TD score that would have occurred if all the teachers in the intervention arm had attended all six TCM training sessions.

Tests of interaction were used in pre-specified exploratory analyses to assess whether or not the effect of TCM on the primary outcome differs across subgroups defined by the following potential moderator variables: school- or child-level deprivation status (in bottom two deciles vs. otherwise), whether or not the child scored in the struggling range on the teacher-reported SDQ-TD score at baseline, the length of the study teacher’s experience (> 5 years vs. ≤ 5 years), KS status (KS 1 vs. KS 2), the child’s gender and cohort status. In a further sensitivity analysis, a test of interaction was used to assess whether or not the effect of TCM on the primary outcome differed between the subgroup defined as having a primary or secondary SEN category of social, emotional or mental health, as classified by the NPD, versus those who did not.

Random-effects Poisson regression was used to compare the pupil-level rates of absence obtained from the NPD between the intervention and TAU arms in years 1 and 2 of the trial. We report crude rate ratios (RRs) and RRs and 95% confidence intervals (CIs) that are adjusted for school-/class-level factors [urban vs. rural/semi-rural area, KS 1 vs. KS 2, deprivation (% of children eligible for free school meals in 2012)], cohort and child gender.

Given the relationship between emotional health and educational attainment,15–18 tests of interaction were also carried out to assess whether or not the effect of the intervention on the assessment of pupil progress (APP) outcomes is modified by whether or not the child scored in the struggling range on the teacher-reported SDQ-TD score at baseline.

Data management

Data entry and cleaning was overseen by the trial manager. All data were stored on a custom-built password-protected database maintained by the Peninsula Clinical Trials Unit (CTU), a UK Clinical Research Collaboration-accredited CTU.

Ethics approval and research governance

This study was granted ethics approval by the Peninsula College of Medicine and Dentistry Research Ethics Committee, now under the auspices of The University of Exeter Medical School Committee on 8 March 2012 (reference number: 12/03/141). The University of Exeter acted as the sponsor for the study. The trial was registered with the International Standard Randomised Controlled Trial Register with the reference number ISRCTN84130388. STARS was hosted in the Child Health Research Group at the University of Exeter, which has experience in the successful delivery of community-based paediatric trials. The trial was overseen by the independent TSC [Paul Stallard (chairperson), Gail Seymour, Shirley Larkin, Tobit Emmens and David Glenny] and the DMC [Paul Ewings (chairperson), Siobhan Creanor and Andrew Richards]. A summary of the changes made to the original protocol57 is given in Table 3, and the complete final trial protocol can be found in Report Supplementary Material 2.

Patient and public involvement

Public involvement has been key at all stages of the design, planning and implementation of the STARS programme of work. A User Advisory Group (UAG) was established comprising parents, teachers, headteachers and behavioural support network teachers who were delivering the TCM intervention. The UAG provided key advice on the acceptability of the study to parents and teachers, recruitment and data collection procedures. Specifically, they provided advice on:

• trial design – the UAG proposed that the TAU schools should be offered the opportunity to receive TCM training as an incentive for recruitment

• costing – the UAG was insistent that the funding for supply teacher cover should be provided as part of the trial as it was considered that schools would be unable to cover these costs themselves and that this would be a barrier to recruitment

• consent procedure – the UAG was fully consulted about the proposed consent process; they commented from their own (teacher/parental) perspective, and the process was adapted as a result of their advice

• outcome measures/questionnaires – the UAG commented on the outcome measures and instructions for completion

• trial literature – the UAG was involved in the development of the trial literature, including information literature and consent forms.

In addition to the UAG, we worked with the school councils of three local primary schools to develop the HIFAMS measure to ask for children’s perceptions of attending school and we asked the children for their views on how it could be adapted to make it more ‘user-friendly’ for children.

Participants

During the 26 months of recruitment, between April 2012 and June 2014, we recruited a total of 80 schools; 40 were allocated to each trial arm across all three cohorts (10 and 5 schools in the TCM and TAU arms, respectively, for cohort 1; 15 and 15 schools in the TCM and TAU arms, respectively, for cohort 2; and 15 and 20 schools in the TCM and TAU arms, respectively, for cohort 3). During the trial some schools did not provide teacher-completed data on child outcomes at the 9-month (n = 1), 18-month (n = 2) and 30-month (n = 1) assessments. In addition, one intervention school withdrew from the trial after completing the 18-month assessment (Figure 3) as a result of a change in headteacher who did not wish to uphold the agreement made by their predecessor. A total of 2075 children were recruited to the trial (1037 in the TCM arm and 1038 in the TAU arm). A further 113 children were either opted out by their parents (n = 107) or ineligible (n = 6). We lost contact with 271 (13%) children over the 30-month follow-up period and two parents withdrew permission for parent-reported outcomes but permitted the collection of teacher- and child-reported outcomes.

FIGURE 3.

Consolidated Standards of Reporting Trials diagram. K, number of schools (clusters); LTFU, lost to follow-up; n, number of children.

Baseline comparability

Compared with the national average,47 participating schools had similar class sizes (trial mean 27.4 vs. national mean 26.8 children) and eligibility for free school meals (trial mean 18.3% vs. national mean 19.3%), but the sample included fewer voluntary controlled schools (5% trial schools vs. 14.4% national schools) and more community (61.3% trial schools vs. 55.3% national schools) and academy schools (10% vs. 6%). Baseline characteristics were generally balanced between the two arms (Table 4). Primary outcome data were collected at 9-, 18- and 30-month follow-up for 96%, 89% and 85% of participants, respectively. The proportion of children scoring in the struggling range on the teacher-reported SDQ-TD questionnaire in both arms approached the expected 20% (cut-off point at the 80th centile)49 but was lower according to parent-reported SDQ-TD (TCM, 16.5%; TAU, 15.5%), which suggests that we lacked parental data on some vulnerable children. No serious adverse events were reported in either trial arm.

Adherence to intervention

Thirty-six (90%) of the 40 teachers in the intervention arm attended four or more TCM sessions; 23 teachers (58%) attended all six sessions (Table 5).

Data completeness

The numbers and percentages of participants with completed data are reported for each time point in Table 6.

Primary outcome

Table 7 summarises the comparison between the trial arms at follow-up for the primary outcome measure. The primary complete-case analysis is labelled CC1 in the table. TCM improved child mental health according to the teacher-reported SDQ-TD score by 1.0 point (95% CI 0.1 to 1.9; p = 0.03) at the 9-month follow-up. There was little evidence, however, of an effect at the 18-month (p = 0.85) and 30-month follow-ups (p = 0.23). The findings from the fully adjusted CC2 analysis were similar, except for the fact that there was only weak evidence of an effect at 9 months on the teacher-reported SDQ-TD (adjusted mean reduction 0.8, 95% CI 0.1 to 1.6; p = 0.09). Post hoc analysis showed that this is because the large number of children lost from the fully adjusted analysis, lost as a result of missing data on the three parent-reported potential confounders, were also those in whom the TCM effect was greatest. The intervention effect on teacher-reported SDQ-TD was –1.6 (95% CI –2.8 to –0.4) for the 534 children with missing data on the three parent-reported potential confounders and –0.8 (95% CI –1.7 to 0.1) for the remaining 1467 children with complete data. Finally, the fully adjusted analysis of imputed data (MI analysis) provided very similar results to our primary partially adjusted analysis (adjusted mean reduction 1.0, 95% CI 0.2 to 1.9; p = 0.02). All of the remaining findings are based on the approach used in the partially adjusted analysis (CC1 analysis). Findings from the CACE analysis were almost identical to those from the primary intention-to-treat analysis, which suggests that the estimated effects would have been no different had all the teachers in the TCM arm attended all six sessions.

Subgroup analysis

Tests of interaction indicated that TCM led to greater reductions in the teacher-reported SDQ-TD score at 9 months (interaction p < 0.001) for children who were classified by their teacher as struggling with their mental health at baseline (mean difference –2.6, 95% CI –4.6 to –0.6) than for children who were not (mean difference –0.4, 95% CI –1.2 to 0.4). A subgroup effect was also found at 30 months (p < 0.001) but not at 18 months (p = 0.10). TCM may also have greater benefits at 30 months (interaction test p-value of 0.02) for children taught by teachers with > 5 years’ experience (mean difference on teacher-reported SDQ-TD score –2.1, 95% CI –3.8 to –0.4) than for children taught by teachers with ≤ 5 years’ experience (mean difference 0.3, 95% CI –1.3 to 1.9). TCM appeared more effective for cohort 2 schools than for cohorts 1 and 3 schools (interaction p-value of 0.02), but there was little evidence of subgroup effects for the other potential moderator variables.

Secondary outcomes

Table 8 summarises the findings from the teacher-reported secondary outcomes. There was evidence, based on the PBQ score, of reduced disruptive behaviour across all three follow-ups (p = 0.04). Likewise, there was evidence that TCM reduces the percentage of children who are classified as struggling according to the SDQ-TD score (p = 0.05) and reduces the Inattention/Overactivity score (p = 0.02) across all waves. At 9 months only, there was also evidence of a reduction in peer relationship problems (p = 0.02) and an improvement in pro-social behaviour (p = 0.02). There was little evidence of effects on teacher-reported Emotions and Impact scores, APP, parents’ assessment of their child’s mental health or the child-reported outcome HIFAMS (Table 9).

National Pupil Database analysis

We received parental consent to access the NPD for a total of 1178 children; this represents 57% of all participating children, with only 71 (3%) parents refusing consent; the remaining 826 parents (40%) did not respond to the invitation.

There was little evidence that the intervention had any effect on the rate of overall absence during either the first (adjusted RR 1.08, 95% CI 0.95 to 1.24; p = 0.24) or second (adjusted RR 1.1, 95% CI 0.72 to 1.7; p = 0.65) year of the trial or on the number of unauthorised absences during the first (adjusted RR 1.03, 95% CI 0.90 to 1.18; p = 0.62) or second (adjusted RR 0.96, 95% CI 0.75 to 1.22; p = 0.74) year of the trial (Table 10). School exclusions were reported on 22 separate occasions (2 in the intervention arm and 20 in the TAU arm), which resulted in a total loss of 64 separate school sessions, morning or afternoon (3 sessions in the intervention arm and 61 sessions in the TAU arm). These exclusions were issued to a total of six children, two from the intervention and four from the TAU arm of the trial. Tests of interaction did not indicate any subgroup effects for children whose primary or secondary category of SEN was social, emotional or mental health, but this analysis lacked power owing to the small number of children with social, emotional and mental health SEN (Table 11).

Sensitivity analysis of pupil progress

Although there was no main effect of the intervention on APP in either literacy or numeracy, subgroup analysis did indicate that the intervention’s effect on APP differs between those who were and were not classified by their teacher as struggling with their mental health at baseline, for both literacy (interaction p = 0.04) and numeracy (interaction p = 0.03). The intervention arm had lower odds than the TAU arm of below-expectation assessments in literacy [odds ratio (OR) 0.77, 95% CI 0.53 to 1.12] and numeracy (OR 0.82, 95% CI 0.59 to 1.14) among those not classified as struggling, but it had greater odds of below-expectation assessments for literacy (OR 1.17, 95% CI 0.70 to 1.94) and numeracy (OR 1.35, 95% CI 0.88 to 2.06) among those who are classified as struggling. All four of these CIs, however, include unity, so it is difficult to interpret these findings, other than to comment that there seems to be a differential effect according to baseline mental health.

Aim

The aim of the economic evaluation was to assess the cost-effectiveness of the TCM course compared with that of TAU over the short- and long-term periods.

Methods

Data analysis

The primary economic evaluation was a within-trial analysis exploring cost-effectiveness at the 30-month follow-up point using the SDQ-TD scores. A secondary analysis using EQ-5D-Y measure of health-related quality of life was proposed, provided that feasibility work to test the EQ-5D-Y in this younger population prior to the start of the trial proved successful. The youth version of the EuroQol 5-Dimensions (EQ-5D) is validated for 8- to 15-year-olds,65 and the study team were testing a simplified version for 4- to 9-year-olds. Our feasibility work suggested that our very young sample struggled to understand the concepts involved, which, combined with a lack of evidence to support the reliability and validity of the EQ-5D-Y among children under the age of 8 years, led us to drop the measure from the protocol. 76

Instead, a published algorithm was used to map SDQ scores to the Child Health Utility Index 9 Dimensions (CHU-9D), a generic preference-based health-utility measure that combines children’s length of life and quality of life into a single summary measure77 and thus allows comparison of interventions in a decision-making context. The CHU-9D was not considered for direct measurement in the study because, at the time of designing and implementing the study, no valuation system was available and the measure was not validated for use with 5- to 6-year olds.

The mapping algorithm, developed in a sample of 200 caregivers of young people in Australia attending CAMHS,78 transforms the five SDQ subscale scores (emotion, conduct, hyperactivity, peer and prosocial) to give utility weights and is presented in Equation 1:

The resulting utility weights are used to calculate QALYs using Equation 2:

So, for example, a child who lives for 2 years with quality of life weighted at 0.5 (2 × 0.5) will have one QALY. QALYs were calculated using the area-under-the-curve approach, in which changes in utility scores were assumed to follow a linear path. 79

Within-trial cost-effectiveness analyses using QALY outcomes were assessed at 9, 18 and 30 months to examine the trajectory of cost-effectiveness in the first instance and to support longer-term decision analysis.

Three scenarios were considered, as per the analysis of clinical data. The base-case analysis (CC1), conducted for within-trial analyses using both SDQ-TD score and QALY outcomes, was based on complete cases and partially adjusted for prespecified potential confounders (cohort, child gender and the three school-/class-level factors used to balance randomisation) and for baseline costs and outcomes, as appropriate. 79 In addition, fully adjusted sensitivity analyses (same confounders as CC1 plus IMD score based on child’s address, number of children living in the household and whether or not the child’s household was rented) were conducted for complete cases (CC2) and 50 multiply imputed data sets (MI) using the chained equations approach for within-trial analysis using SDQ-TD scores.

Results

Short-term cost-effectiveness analysis using Strengths and Difficulties Questionnaire Total Difficulties score

For the primary cost-effectiveness analysis using the SDQ-TD score, the lower costs and better outcomes in the intervention group generate an ICER of –£19.90 per unit improvement in SDQ-TD, which suggests that the intervention dominates TAU and is cost-effective at the 30-month follow-up point.

Figure 4 shows the scatterplot of bootstrapped mean differences in costs and outcome. The majority of the scatter points indicate that TCM is more effective than TAU (to the right of the y-axis) and lie in the north-east quadrant (50%), where TCM is more effective but more costly, and the south-east quadrant (29%), where TCM is more effective and less costly. The remaining scatter points show poorer outcomes for TCM than for TAU and fall in the north-west (14%; less effective, more costly) and south-west (7%; less effective, less costly) quadrants.

FIGURE 4.

Bootstrapped mean differences in costs and effects (SDQ-TD score) of intervention compared with TAU at 30-month follow-up (CC1). NE, north-east; NW, north-west; SE, south-east; SW, south-west.

The CEAC for the partially adjusted primary analysis (SDQ-TD score; CC1) suggests that the probability of TCM being cost-effective compared with TAU ranges from just under 40% at a zero willingness to pay for a unit improvement in SDQ-TD score to nearly 80% at a £1500 willingness-to-pay threshold (Figure 5). The point at which TCM has a higher probability of being cost-effective compared with TAU (i.e. probability > 50%) is at a willingness-to-pay level of £70 per unit improvement in SDQ-TD score.

FIGURE 5.

Cost-effectiveness acceptability curve showing the probability that TCM is cost-effective compared with TAU for different values of willingness-to-pay thresholds using the SDQ-TD score at 30-month follow-up (CC1).

The cost and SDQ-TD results of the sensitivity analyses using fully adjusted (CC2) and imputed (MI) data sets are presented in Table 17. There were no statistically significant between-group differences in costs or outcomes for either of the sensitivity analyses. For CC2, AMD of total costs (£16.44, 95% CI –£155.32 to £188.21; p = 0.9) and outcomes (–0.5, 95% CI –1.7 to 0.6; p = 0.4) were similar to that of the primary analysis (CC1), supporting the high probability of TCM being cost-effective for a range of willingness-to-pay thresholds per unit improvement in SDQ-TD score. Imputation of missing data (MI) showed a larger difference in both costs (AMD £252.00, 95% CI -£221.00 to £725.00; p = 0.3) and outcomes (AMD –0.6, 95% CI –1.6 to 0.5; p = 0.3) and had a negative impact on the probability of TCM being cost-effective, but only at the lower levels of willingness-to-pay thresholds (Figure 6).

TABLE 17 - Mean costs (£) and SDQ-TD scores per participant over the 30-month follow-up period for sensitivity analyses (CC2 and MI)

Outcome	Intervention, mean cost, £ (SE)	TAU, mean cost, £ (SE)	CC1 AMD (n = 1007)			CC2 AMD (n = 1007)			MI sensitivity analysis AMD (n = 2075)
			Estimated	95% CI	p-value	Estimated	95% CI	p-value	Estimate	95% CI	p-value
Total costs	524.30 (91.75)	528.67 (67.34)	30.24	–140.98 to 201.47	0.7	16.44	–155.32 to 188.21	0.9	252.00	–221.00 to 725.00	0.3
SDQ-TD score	5.2 (0.6)	5.4 (0.5)	–0.5	–1.7 to 0.6	0.4	–0.5	–1.7 to 0.6	0.4	–0.6	–1.6 to 0.5	0.3

FIGURE 6.

Cost-effectiveness acceptability curves showing the probability that TCM is cost-effective compared with TAU for different values of willingness-to-pay thresholds using SDQ-TD score (all analyses).

Short-term cost-effectiveness analysis using quality-adjusted life-years

For the secondary cost-effectiveness analysis using QALYs, the poorer outcomes and higher costs suggest that the intervention is dominated by TAU at the 30-month follow-up point and is not cost-effective (Figure 7). The majority of scatter points lie in the north-west quadrant (61%) and indicate that TCM is less effective and more costly; 34% of scatter points fall in the south-west quadrant, showing that TCM is less effective and less costly, and the remaining scatter points fall in the north-east (3%; more effective, more costly) and south-east (2%; more effective, less costly) quadrants.

FIGURE 7.

Bootstrapped mean differences in costs and effects (QALYs) of intervention compared with TAU at 30-month follow-up (CC1). NE, north-east; NW, north-west; SE, south-east; SW, south-west.

The CEAC for the short-term cost-effectiveness analysis using QALYs suggests that the probability of TCM being cost-effective compared with TAU at 30 months is < 10% at the NICE willingness-to-pay threshold of £20,000 per QALY (Figure 8).

FIGURE 8.

Cost-effectiveness acceptability curve showing the probability that TCM is cost-effective compared with TAU for different values of willingness-to-pay thresholds using QALYs at 30-month follow-up (CC1).

To explore changes over time, cost-effectiveness using QALYs was also explored at 9 and 18 months in post hoc analyses.

The scatterplot of bootstrapped mean differences in costs and QALYs at 9-month follow-up indicates that TCM is more effective than TAU (Figure 9). The majority of the scatter points lie in the north-east quadrant (87%), where TCM is more effective but more costly. The very few remaining scatter points show poorer outcomes for TCM than for TAU and fall in the north-west (8%; less effective, more costly) and south-west (1%; less effective, less costly) quadrants.

FIGURE 9.

Bootstrapped mean differences in costs and effects (QALYs) of intervention compared with TAU at 9-month follow-up (CC1). NE, north-east; NW, north-west; SE, south-east; SW, south-west.

The CEAC for cost–utility at 9-month follow-up suggests that the probability of TCM being cost-effective compared with TAU is 60% at a willingness-to-pay threshold of £20,000 (Figure 10).

FIGURE 10.

Cost-effectiveness acceptability curve showing the probability that TCM is cost-effective compared with TAU for different values of willingness-to-pay thresholds using QALYs at 9-month follow-up (CC1).

Similarly, the scatterplot of bootstrapped mean differences in costs and QALYs at 18-month follow-up indicates that TCM is more effective than TAU (Figure 11). The majority of the scatter points lie in the north-east quadrant (40%), where TCM is more effective but more costly, and the south-east quadrant (41%), where TCM is more effective and less costly. The remaining scatter points show poorer outcomes for TCM than for TAU and fall in the north-west (10%; less effective, more costly) and south-west (9%; less effective, less costly) quadrants.

FIGURE 11.

Bootstrapped mean differences in costs and effects (QALYs) of intervention compared with TAU at 18-month follow-up (CC1). NE, north-east; NW, north-west; SE, south-east; SW, south-west.

The CEAC for cost–utility at 18-month follow-up suggests that the probability of TCM being cost-effective compared with TAU is nearly 80% at a willingness-to-pay threshold of £20,000 (Figure 12), which is considerably higher than the results at 30-month follow-up (< 10%).

FIGURE 12.

Cost-effectiveness acceptability curve showing the probability that TCM is cost-effective compared with TAU for different values of willingness-to-pay thresholds using QALYs at 18-month follow-up (CC1).

Table 18 shows a summary of costs, QALYs and cost-effectiveness over the follow-up period. At a willingness-to-pay threshold of £20,000 per QALY, the probability of TCM being cost-effective is 61% at 9 months and 80% at 18 months but drops to 5% at 30 months. This summary trend is illustrated in Figure 13 and shows that the AMD in costs (TCM – TAU) increases over time, whereas the AMD in QALYs (TCM – TAU) decreases, which has an impact on the probability of TCM being cost-effective after 18 months. The figure also presents trend lines for differences in costs and effects that assume a linear relationship with time and show one possibility of how changes may have occurred over the trial period. The cost-effectiveness analysis does not depend on this assumption, so the linear relationship is not tested.

TABLE 18 - Summary of costs, QALYs and cost-effectiveness over time

N/A, not applicable; WTP, willingness to pay.

Summary	Time point
	Baseline	9 months	18 months	30 months
Costs (£)
AMD (TCM – TAU)	–11.67	39.07	–1.08	30.24
95% CI	–49.90 to 26.55	–6.08 to 84.21	–51.31 to 49.15	–140.98 to 201.47
QALYs
AMD (TCM – TAU)	–0.001	0.003	0.004	–0.05
95% CI	–0.006 to 0.004	–0.001 to 0.01	–0.003 to 0.01	–0.11 to 0.01
Cost-effectiveness
Probability that TCM is cost-effective at WTP of £20,000 (%)	35	61	80	5
Cost/QALY when TCM > TAU (£)	N/A	16,000	10	N/A

FIGURE 13.

Summary of costs and outcome over follow-up.

Discussion

Although the results of the clinical analysis alone do not support the hypothesis that TCM is more effective than TAU, the primary cost-effectiveness analysis, comparing outcomes in terms of total SDQ-TD scores, suggested that TCM may be cost-effective compared with TAU at 30 months for a relatively wide range of values of willingness to pay for improvements in SDQ-TD score. This is due to the economic analysis being concerned with the joint difference in both costs and effects; costs were slightly lower and effects (SDQ-TD scores) were slightly better in the TCM group than in the TAU group. It is also important to note that although the clinical analysis focused on statistical significance, the recommended decision-making approach for economic evaluation is to explore cost-effectiveness irrespective of statistical significance in order to make the best use of the available evidence and to determine which intervention has the higher probability of being cost-effective. 85,86

The economic results using the SDQ-TD were not altered in the sensitivity analysis controlling for additional randomisation variables (CC2) but were affected by missing data (> 50% missing in CC1). The sensitivity analysis with MI data showed that the probability of TCM being cost-effective was reduced at lower willingness-to-pay thresholds (< £1500) for a unit improvement in total SDQ-TD scores. Although the SDQ is a commonly used measure of mental health in children and adolescents,36 it does not have a set of utility weights and is, therefore, of limited use in allocating scarce health-care resources. With very small differences in costs and effects, as in this study, it is not possible to draw firm conclusions about the cost-effectiveness of TCM without knowing society’s willingness to pay for improvements in SDQ-TD scores.

The development of mapping algorithms enables researchers to translate information gained from a non-preference-based measure (such as the SDQ) into health state utility weights for use in economic evaluations. In settings or populations in which it may not be possible to ascertain direct evidence on health state utility (e.g. in primary school children), mapping is considered the only feasible way to conduct cost–utility analyses. One such mapping algorithm, estimated in Australia using the CHU-9D, was used to estimate utility scores for a secondary within-trial analysis using QALYs as the measure of effect, following poor performance of the EQ-5D-Y in feasibility testing.

Mapped utility scores changed very little between trial arms and over time, and the mean score at 30-month follow-up was close to 0.88 in both groups, which approximates the weighted average mean utility score for a general population sample aged < 12 years from developed countries in health-care and school settings. 87 The mapped cost–utility analysis results suggest that TCM has a higher probability of being cost-effective than TAU at 9 and 18 months but not at 30 months, which raises questions of the appropriate time horizon for analyses of this type. Although health economic evaluations tend to focus on the longest time horizon available within a trial in order to provide the maximum opportunity for cost and outcome impacts to be recorded, in trial-based evaluations it is conceivable that the majority of evaluations demonstrating cost-effectiveness of interventions at early follow-up points would eventually reach a point at which cost-effectiveness was no longer evident, as the time horizon was extended, primarily because of the effects of interventions ‘wash-out’ over time.

Although NICE recommends the use of mapped health state utility estimates when directly collected data are not available,73 the validity of these mapping methods has not been fully addressed. Mapping can result in a loss of information and increased uncertainty and thus requires a closer assessment of the agreement between target and source instruments. It has been argued that current methods for mapping are not known to be conceptually robust88 and carry a ‘significant risk that may be harmful to population health’. 89 It is therefore important to consider the results of the mapped cost–utility analysis with greater uncertainty. It is possible that vital information is lost when mapping the five subscores of the SDQ (emotional symptoms, conduct problems, inattention/hyperactivity, peer problems, and a pro-social scale) to the nine dimensions of the CHU-9D (sad, worried, pain, annoyed, tired, homework or schoolwork, daily routine, activities and sleep) and could therefore lead to erroneous conclusions about cost-effectiveness. Authors of the mapping algorithm used in the present study noted that although the algorithm performed well in predicting mean group observed utility values, it did not perform well in predictions at individual participant level. 78 Despite published guidelines and a checklist for the reporting of mapping algorithms,90,91 only a few studies have been found to fulfil or partly fulfil all criteria on the checklist. 92

Small differences in SDQ-TD and mapped utility scores over the trial follow-up made extrapolation beyond the trial period unnecessary, as the baseline beyond which data would be extrapolated from the end of the trial period was almost identical in the two groups. Longer-term analysis would therefore serve only to replicate the within-trial results. Although small clinical differences can result in large changes in health outcomes over longer-term models, there was limited evidence to suggest this in the trial data, particularly given the sharp drop in the probability of TCM being cost-effective using QALYs from 79.5% at 18 months to 4.6% at 30 months. It is, therefore, fair to assume that any projections of costs and effects beyond 30 months would be of questionable validity. Appropriate temporal extrapolation of costs and utilities is reliant not only on clinical outcomes but also on many other long-term processes, such as adaptation to particular health states, the data on which are not currently available for children and adolescents.

Finally, the study was limited by failing to consider spillover effects in the quality of life of teachers. TCM has been designed to increase the well-being of teachers, as well as of students, and future research should attempt to capture these effects by directly administering a preference-based quality-of-life instrument such as the EQ-5D, which is commonly used in adults. 93

Overview

The STARS trial used a variety of IY-generated pre-existing materials to ensure that the course was delivered with fidelity to the TCM model, which included the use of manualised checklists of the topics covered by each session and monthly supervision with the programme developer. All group leaders attended the required 3-day basic training in TCM and delivered two courses each as part of the feasibility work to ensure their familiarity with the manual. 94 There is no existing tool to assess independently how faithfully group leaders deliver training, so, as part of this trial, we began the process of developing such a tool. In addition, we used a pre-existing observational measure, the Teacher–Pupil Observational Tool (T-POT),42 to assess if the TCM training had altered the behaviour of teachers in their classrooms. These different methods of assessing fidelity, and their findings, are described in turn below. The data reported in this chapter include only the six TCM groups that included intervention teachers. The final two TCM groups delivered in Year 4 of the study, which included only TAU teachers, are not represented in this chapter.

Supervision with programme developer

Throughout their time delivering TCM workshops for the STARS trial, the six group leaders were supervised by the TCM programme developer, Carolyn Webster Stratton, to ensure fidelity to the TCM model. All TCM workshops were video-recorded for use as part of the supervision process. After each workshop, group leaders from each TCM group selected a 10- to 30-minute video section of the day (either one long clip or a variety of shorter clips) to be reviewed by Carolyn Webster Stratton. The videos could be examples of a section that teachers felt went well, or a section that they felt did not go as well and on which they wanted advice, which they explained in a video review form. These materials, plus the teachers’ evaluations of that workshop, were sent to Carolyn Webster Stratton, who returned a detailed review of the video clips with comments about how to improve the delivery of the course; these reviews were subsequently discussed in a 1-hour video conference call with all six group leaders. This method of supervision began during the 2 years of feasibility work, which allowed time to refine the process.

The supervision process was very successful and Carolyn Webster Stratton was confident that the group leaders were delivering the course with fidelity to the model. The process was also well received by all of the group leaders who, despite competing time pressures, always completed the necessary preparation.

Group leader-completed agendas and checklists

As part of the supervision process, group leaders completed standardised ‘agendas and checklists’ of the expected curriculum that they covered in each workshop. These were developed by the TCM programme developers to report on those ‘activities’ that they had delivered during the session. The checklist included information on:

• training vignettes that were shown in the workshop

• group activities that were used in the workshop, such as specific role-plays or buzz ideas, explaining the rationale for different TCM techniques

• basic housekeeping expected when running the TCM training, such as writing up the day’s agenda and ensuring that the teachers completed their self-monitoring checklist, which was completed after each session.

The checklists and agendas were an important tool for ensuring fidelity to the model, as the video review supervision discussed in the section above was only a small segment of a workshop that lasted the whole day. It was, therefore, important to have a method to assess the extent to which important elements of the day’s workshop were completed.

Analysis of the ‘agendas and checklists’ from the six courses found that the mean number of digital versatile discs (DVDs) shown per course group ranged from 14.8 to 19.5. The group leaders tended to show more DVDs in the first two workshops (workshop 1: building positive relationships with students, and the proactive teacher, mean videos shown = 24.9; workshop 2: teacher attention, coaching, encouragement and praise, mean videos shown = 28.9), whereas considerably fewer were shown in workshop 5 (decreasing inappropriate behaviour – follow through with consequences, mean videos shown = 7.8; Table 19). This difference is consistent with the number of DVDs that were available for group leaders to show in these workshops (70 for workshop 1, 57 for workshop 2 and 21 for workshop 5).

The number of DVDs shown in each workshop was fairly consistent across groups, with the exception of Group 5, which was shown slightly fewer DVDs in workshop 2 (n = 15), and Group 1, which was shown fewer in workshop 3 (n = 8) (see Table 19). This may be because the group leaders had decided that they needed more time for role-play and discussion.

Group leaders were required to cover certain activities during each workshop and mark whether or not they had done so using a simple checklist. Group leaders reported doing the majority of these activities (mean 92% of activities; range 73–100%) (Table 20).

Development of new fidelity tool

The supervision process with the IY developer was very successful and demonstrated faithful running of the course; however, we lacked a tool to make independent assessments of fidelity. Using TCM recorded sessions, we developed a quantifiable measure that could be used in the future to assess fidelity formally.

Given that all of the TCM workshops in the STARS trial were video-recorded, this was the most suitable, and rich, source of data to use to construct the measure. The observational measure was designed to systematically analyse specific segments of these video clips, identifying what group leaders did using a tally sheet to code predefined behaviours/actions. Owing to time constraints, only one TCM topic could be studied, and the selected topic needed to be challenging, operationalised and to have a large proportion of the topic delivered by the group leaders.

The element of the course that best met these principles was the TCM delivery of time-out. We had observed that both teachers and the group leaders may initially struggle with the concept of time-out. Time-out in the TCM training is not a punishment but rather a method of giving a child time to calm down in order to regain emotional control to avoid negative behaviours; time-out is the most operationalised strategy in the TCM training. It is also pivotal because if it is not delivered well, and in line with the intervention, it has the potential to undermine the whole TCM training and the principles underpinning it.

An initial tally sheet of behaviours and actions that would be expected from group leaders was developed and trialled using TCM groups delivered in our feasibility work. 94 It is important to note that, as with all interventions, group leaders may tailor elements of the programme to meet the needs of each of the specific groups, provided that core components are delivered. 23

Based on the experience of applying the tally sheet to feasibility groups, we reduced the number of items to tally (removing duplicates or streamlining items), changed the layout of the sheet so that it flowed better and was easier to use, and edited the descriptions of the items so that they were more concise. The tally sheet was finalised in September 2016 (Figure 14) and contained 24 items based on the principle of time-out within TCM. The included items were all chosen for theoretical reasons. There were three sections:

1. explanation of time-out – including whether or not group leaders used questions and discussions and created a supportive atmosphere

2. general – this included free text on what methods were used to explain time-out to teachers and children (e.g. role-play, videos), what activities were used and if the group leaders displayed certain behaviours such as asking for specific feedback

3. principles of time-out – this was the more theoretical section of the tally sheet, which aimed to identify if the group leaders explained the principle with fidelity to the IY model and also acted as an aid for those who may not be familiar with the principle of time-out.

FIGURE 14.

The final tally sheet used to assess the fidelity of the delivery of TCM time-out. TO, time-out.

Once the tally sheet had been finalised, two researchers independently coded the session from Group 1 of the STARS trial and, once coding rules were agreed, good agreement was reached between the coders. In total, videos from the time-out section of five TCM courses in the STARS trial were assessed using the fidelity tool because of a failed recording of one course (Group 4). The median length of the videos on time-out was 1 hour 57 minutes (IQR 103–133 minutes). As shown in Figures 15 and 16, the tool was sufficiently sensitive to detect differences between the TCM groups in terms of different aspects covered by this section of the TCM training that triangulate with other data availability on the process of delivery. For example, group leaders from Group 5 sought advice in supervision about troublesome group dynamics and teacher feedback was less positive than in other groups. The raters were blind to these issues when completing the tally. The observed differences in frequencies may be because of differences in the personalities of the group leaders, the dynamics of each particular group and how quickly the group understood the principles not only of time-out but also of the whole TCM training.

FIGURE 15.

Graph of the frequency with which the group leaders highlighted principles of time-out in the five TCM courses (from explanation of time-out section of the tally sheet).

FIGURE 16.

Graph of the frequency with which group leaders showed empathy in the five TCM courses (from supportive atmosphere section of the tally sheet).

Using the video data from the STARS trial, we have a prototype measure that could be applied in the future by different researchers to assess the fidelity of a group leader’s delivery of the most challenging session within the TCM training. We acknowledge that the measure requires additional testing and benchmarking against other TCM training groups. There are also limitations of the measure that need to be addressed before the tool can be used more widely. The next stage in the development of the model is to develop more precise rules for using the measure and to test the reliability and replicability of the tool. Once complete, we hope that the measure will be used by other researchers when analysing the TCM training.

Teacher–pupil Observation Tool

Discussion

All intervention trials must measure fidelity to the model to ensure that the trial is truly evaluating the intervention as intended. Fidelity encompasses adherence to, exposure to and quality of the intervention;100 all aspects should be assessed. In the STARS trial, a number of pre-existing fidelity tools were used: the ‘agendas and checklists’, high-quality training and regular supervision with the TCM developer to measure adherence and exposure to the intervention. Supervision with the programme developer was very successful and it was telling that her feedback was positive in terms of the fidelity and quality of delivery of the course. We used video data from the STARS trial to develop a tool that could be applied in the future by different researchers to assess the fidelity of a group leader’s delivery of the most challenging session within the TCM training. The measure could be used to explore variability within delivery in future studies.

The impact of the TCM training on teachers’ behaviour is also an important factor in the fidelity of the intervention. To assess this, we used the pre-existing T-POT measure, an observational tool, with independent observers rating teachers’ behaviour in the classroom. Results from the T-POT show increased use of labelled praise as expected, but, counter to our hypothesis, increased indirect commands by teachers in the intervention group compared with teachers in the TAU group. Teachers in the intervention group also used more positive and less neutral behaviours than the TAU group teachers at follow-up, which was predicted. These results suggest changes in the teachers’ behaviour after attending the TCM training, but ideally observations would have been carried out in all schools and at all data points.

The endorsement of the programme developer that the programme was delivered with fidelity and to high quality, combined with the group leaders’ completion of the TCM programme’s checklists and agenda, means that we can be confident that teachers had access to the training as intended. Observations via the T-POT suggested that access to TCM predicted some, but not all, of the predicted changes in teachers’ behaviour. This analysis lacked power, as not all classrooms were observed and future studies should observe all classrooms. Our measure of fidelity revealed some variance between groups on the most contentious and challenging aspect of the TCM course, but, with an invalidated measure and only six groups, we were not able to pursue it further. Additional research is needed to ensure that the fidelity tool we have developed is reliable and replicable, at which point studies could be pooled to see if variance in delivery mediates variability in outcome.

Background

The process evaluation aimed to explore teachers’, headteachers’ and TCM group leaders’ experiences of TCM, the use and uptake of TCM strategies in the classroom by teachers and other staff and influences on this, and also aimed to improve our understanding of the main outcomes more generally. Our main research questions included:

1. What are the key elements of the TCM training from a teacher’s perspective?

2. What are the barriers to and facilitators of the use of TCM strategies in the classroom?

3. What do teachers feel has changed in their approach? What impact has it had on others?

4. Which TCM strategies have been used/not used?

To address these questions, both qualitative and quantitative methods were used (see Hansford et al. 101) for the process evaluation protocol. These methods included:

1. teacher focus groups (FGs) interviews

2. headteacher interviews

3. group leader FGs/interviews

4. Teacher Satisfaction Questionnaire (TSQ).

This chapter describes the above methods in more detail, presents the results and discusses these in the context of the process evaluation. Although considered part of the process evaluation, information about fidelity is discussed in Chapter 5.

Methods

Results of teacher, headteacher and group leader focus groups/interviews

Participant details

A total of 80 schools were given the opportunity for one teacher to attend TCM training. Intervention teachers (n = 40) received TCM training in their first year of involvement in the study and TAU teachers (n = 40) were offered a place on the TCM course 1 year later, when they were no longer teaching the trial children.

Of the 80 schools given the opportunity, four TAU schools chose not to send a teacher on the training: two because the original trial teacher had left the school and two because the original trial teacher was deemed too busy to attend. Therefore, in total, 76 schools accessed the training course. One TAU school chose to send a different teacher on the TCM training, as the original teacher had taken on extra responsibilities and did not have the time to attend that year; however, this replacement teacher attended only two sessions, feeling that the course was too basic for them, and later left the school. We therefore offered the training place to the original teacher the following year, in their final year of involvement in the trial. Figure 17 shows how many teachers accessed the training and in which year of involvement in the study this occurred.

FIGURE 17.

Number of teachers who attended the TCM training in each year.

The process evaluation was based on the 77 teachers who attended the TCM training and the respective 76 schools at which these teachers taught.

Results and discussion

Below are the key themes that emerged for each research question. Additional examples of each theme/subtheme can be found in Appendix 4. The source of quotations is identified as originating from teachers, group leaders or headteachers, and is followed by an indication of whether the quotation was obtained during an interview or FG. Interviews and FGs are identified by number so readers are able to get a sense of how many individual participants expressed particular themes. However, care has been taken to ensure that it is not possible to identify any participant or school from the content of the quotation.

Influences on teachers’ learning: what were the key elements of the Teacher Classroom Management training from the teachers’ perspective?

Teachers’ and group leaders’ feedback revealed a number of factors that influenced teachers’ learning on the TCM course. Headteachers, however, did not comment on this, although one mentioned that sharing may have been important. Lack of feedback from headteachers may be a result of teachers not discussing this element of TCM with them.

Barriers to and facilitators of the use of Teacher Classroom Management strategies in the classroom

Teachers described a number of factors that either facilitated or hindered their implementation of TCM strategies within their own classrooms. Some of these related to the course itself and others related to the contextual situation in which the teacher was attempting to implement the strategies. Group leaders and headteachers did not describe any facilitators, although they did touch on a few potential contextual barriers.

Facilitators

Teachers reported that understanding the theory underpinning the strategies gave them a more informed and structured approach to behaviour management. It encouraged them to think more about both what they were doing and why, and also what the child was doing and why:

I’ve had lots of challenging children in my class over the last few years [. . .] and Behaviour Support have come in or we have worked with ways in school, but this has joined all the dots really. You might get half an hour of Behaviour Support person going ‘OK try this, try this, try this’. This for me has been the background to all that they are saying which has made it make sense [. . .] so it has definitely, you know, filled in the gaps that I had.

Teacher, FG 5

Teachers particularly commented on the TCM pyramid (Figure 18) as a useful graphic to illustrate the principles underlying the TCM approach:

It’s a nice ethos to think, don’t get to the top, try and think of everything you can possibly do before you reach that ultimatum, so it does just make you stop and think about the ways that you can organise your class and the ways that you can respond to the children before you reach that, that top of the pyramid.

Teacher, FG 2

FIGURE 18.

Teaching Pyramid^®. © The Incredible Years^® Inc. and Dr Carolyn Webster-Stratton. Reproduced with permission.

©The Incredible Years^® Inc. and Dr Carolyn Webster-Stratton. Reproduced with permission.

Teachers liked the practical nature of the strategies for managing behaviour and building relationships, and the fact that they were ‘easy to use’ for the whole class while also meeting individuals’ needs. Teachers liked the fact that they had a range of strategies at their fingertips:

We’ve got so many more tools in our box, and it’s just so much easier, you know if one strategy you try doesn’t work with a child you’ve got a plethora of other ones you can have a go at.

Teacher, FG 1

They also talked about the benefit of learning to adjust and fine-tune strategies in order to suit the context, whether this be the child’s needs, the school environment or just the fact that ‘no one thing will work forever’.

Teachers liked the fact that, as well as recognising that different strategies may work with different children, there was an acknowledgement in the way the course was taught that different techniques may or may not suit an individual teacher’s approach or personality.

What do teachers feel has changed in their approach? What impact has it had on others?

Impact on children

Impact on parent–teacher relationships

Developing relationships with parents is advocated by the TCM approach, and teachers mentioned various ways in which they had consciously tried to do this, for example by greeting parents at the door, going out to see them in the playground to pass on positive messages about what their child had done, setting up a home-school book, sending positive letters and making telephone calls. One headteacher also noticed this:

I mean he was thinking that he was always listening and welcoming parents [. . .] but he wasn’t so much seeking out opportunities to share positive moments with parents but he feels he does that more now and that parents really appreciate it. And at the end of the day, especially if your child has been previously or in the past been causing problems or having difficulties with learning, that it’s nice to be actually told actually he or she is doing really well and trying hard. So he felt he does that a lot more and can feel the sort of appreciation on behalf of the parents when he does that.

Headteacher, interview 22

Some teachers talked about how their efforts to develop better relationships had resulted in parents perceiving the teachers more positively, prompting more positive responses and more discussion:

I have noticed that there has been a much more positive parental response. Because when they see you now they feel they can actually talk to you, more rather than just listening to a litany of things that little Johnny has done wrong today.

Teacher, FG 2

One teacher described how this positive effect had been recognised more widely within the school, and that the whole school was now working towards sending more positive messages to parents, dealing with negatives in a ‘gentle’ way, and generally encouraging parents to participate more actively in school life:

It’s definitely impacted parents which has been a major culture shift, for parents to feel a positive response or message. And actually, our school is driven to work better with parents and open the doors and become more of an open door policy, to encourage children to come in and settle with parents alongside.

Teacher, interview 05

Some teachers had been approached by parents to help them try to tackle difficult behaviour at home using some of the strategies that they had adopted in the classroom. One teacher talked about advising a parent to focus on positive instead of negative behaviours. Others talked about how this new co-operative way of working with parents had the added benefit of not only helping parents but also introducing consistency and therefore reinforcing behaviour changes at school:

Dad initially was very, quite aggressive, now whenever he sees me it’s all smiles and ‘oh yes’, they have implemented the same strategy we have at school at home – the traffic light system, you know, for behaviour – and we have noticed a complete difference in him. He still does the odd thing [. . .] but there has been a notable change in his attitude to school and also in the amount of work he is doing.

Teacher, FG 2

Another headteacher observed that, in combination with support from the school, the course had helped highlight that the teacher needed to engage more with parents and talk to them positively. The teacher had started to do this, although the headteacher thought that there was still more work to do.

Use of Teacher Classroom Management strategies

The techniques and approaches described below are those that teachers commonly mentioned. Group leaders did not comment on what specific strategies teachers were now using and, although some headteachers mentioned specific strategies that their teacher had been using, many seemed unaware of the strategies teachers had employed, if any.

Written feedback completed by the teachers

Results and discussion

Overall, the majority of teachers rated content, videos, teaching and discussion as either ‘helpful’ or ‘very helpful’ (Figure 19).

FIGURE 19.

Percentage of teachers who found TCM ‘helpful’ or ‘very helpful’. Teacher feedback on course.

Most teachers thought that the content, teaching and discussion were ‘very helpful’ (58%, 77% and 83%, respectively) and that the videos were ‘helpful’ (61%). Videos received more ‘not helpful’ or ‘neutral’ ratings than the other three areas (2% and 12%, respectively). Comments from the teachers suggested that they did find the videos useful to support discussions, but they thought that they were not always culturally relevant to the classrooms in which they teach. This ties in with our findings from the FGs/interviews.

When studying the teacher feedback by each TCM workshop, teachers rated both the teaching and discussion consistently as either ‘helpful’ or ‘very helpful’ across all six workshops. Teachers also rated the content highly across the workshops, although ratings for this dropped slightly in workshops 5 and 6. These two workshops cover topics that could be considered more challenging (see Table 1 for an overview), which may explain this slight decrease in favourable content ratings. The percentage of teachers who rated the video clips as ‘helpful’ or ‘very helpful’ was high across all workshops (although lower than for the other three TCM areas) and appeared to be lower in workshops 2 and 3 (Figure 20). Teachers may have thought that the behaviours and strategies covered in these workshops were more ‘obvious’ and may have felt that they did not need to see as many video clips as a result, valuing other aspects of the workshop more.

FIGURE 20.

Percentage of teachers who rated course content, video clips, teaching and discussion as ‘helpful’/’very helpful’. Teacher feedback on course by TCM workshop.

When examining feedback by TCM group attended (i.e. TCM groups 1–8), there appears to be greater variation in the percentage of teachers rating each area favourably (Figure 21), suggesting that there may have been a difference in how teachers experienced the course depending on what year/group they attended. The greatest variation is found in terms of whether or not the videos were helpful, with them being noticeably less helpful in the final years of the study. This variation in helpfulness is likely to be due to the group dynamic and how well the teachers in that group responded to the video clips, which were often considered outdated and ‘Americanised’, as discussed previously (see Materials).

FIGURE 21.

Percentage of teachers who rated the course groups as ‘helpful’ or ‘very helpful’. Teacher feedback on course by course group.

On average, there was little variation in how useful teachers found, and how frequently they used, the TCM teaching strategies on the TSQ. Immediately post course, teachers reported finding the strategies useful (mean 4.25, SD 0.27) and using them frequently (mean 4.05, SD 0.04). This corresponds with the findings from the qualitative work, which suggested that teachers were employing a range of strategies from the TCM training and finding a number of these strategies useful.

Overall discussion

The process evaluation aimed to address four main questions:

1. What are the key elements of the TCM training from a teachers’ perspective?

2. What are the barriers to and facilitators of the use of TCM strategies in the classroom?

3. What do teachers believe has changed in their approach and what impact has it had on others?

4. Which TCM strategies have been used/have not been used?

We explored these questions through FGs/interviews with teachers, headteachers and group leaders and also through questionnaires designed to examine teachers’ satisfaction with the TCM training.

Overall, qualitative results suggested that teachers mostly enjoyed the TCM training and found it beneficial. This was echoed in the TSQ, which suggested that the majority of teachers found the TCM training either ‘helpful’ or ‘very helpful’.

In relation to our first research question, ‘what are the key elements of the TCM training from a teachers’ perspective?’, qualitative findings suggested that the structure of the course (i.e. being 1 day a month over a 6-month period), being part of a group that came together to discuss practice and learn from one another, having time out of the classroom to ‘reflect’ on practice, and being able to ‘see, practice and implement’ the strategies back in the classroom were key elements. The TSQ highlighted that teachers experienced aspects such as the course content, video clips, discussion and the delivery of the TCM training either as ‘helpful’ or ‘very helpful’.

In relation to our second research question, teachers reported limitations in the course content or obstacles within their school settings (e.g. a lack of a ‘whole-school’ approach, lack of support from colleagues/senior staff members, clashes with school policy, and external bodies such as Ofsted) as their main barriers to the implementation of TCM strategies in the classroom. Although often a widespread view,94 only one teacher reported the TCM training as not being as useful for KS 2 children as a whole. Facilitators, however, included understanding the theory behind the strategies (something endorsed by the TCM training), the adaptability/flexibility of the TCM strategies and the fact that the strategies were generally easy to use.

Teachers mentioned using a number of different strategies as a result of the TCM training, including specific and proximal praise, ignoring, rewards and consequences, time-out, emotional regulation and behaviour plans. They also reported attempting to use coaching and commentary, although the feedback on this was more mixed. The TSQ complemented these findings, suggesting that teachers found the strategies promoted by TCM useful and reported using them frequently.

Teachers suggested that the TCM training had affected them in a number of ways, which included changing their mindset; helping them to see things from the child’s point of view; helping them to develop better relationships with the children; making them think more before responding; helping them to feel more positive, more confident and more in control; and helping them to create a more positive cycle of behaviour through modelling. These impacts were all extremely positive. They also believed that TCM had affected the children in that they were calmer, more motivated and more ready to learn; however, the views on whether or not the strategies had had an impact on children with behavioural needs were more mixed. A few teachers also reported that TCM had had an impact on their relationships with parents.

Introduction

Teaching is commonly acknowledged to be an extremely stressful occupation, with poor and possibly worsening retention in the UK. 103 The prolonged response to chronic emotional and interpersonal stressors in the workplace can lead to ‘burnout’, a syndrome of exhaustion, cynicism and reduced professional efficacy, which may underlie poor mental health and low retention among teachers. 104 Teachers often report a lack of training in, and difficulties with, managing disruptive behaviour, which is associated with stress, burnout and exit from the profession. 11,12,105 Jennings and Greenberg106 proposed a ‘burnout cascade’, whereby a teacher’s difficulties with behavioural management in the classroom lead to a decreased sense of self-efficacy, which results in negative consequences for their relationship with the children, a more reactive and more negative classroom environment, and increasing distress and burnout for the teacher.

In general, mental ill health is both the largest single cause of disability and the leading cause of sickness absence from work in the UK. 107 The economic impact of poor mental health occurs not just through lost working days and absenteeism but also through the reduced productivity of employees still present at work, known as ‘presenteeism’. The education and care sectors have the highest rates of presenteeism of any employment sector, which in turn predicts high levels of absenteeism. 108 Several studies report high rates of depression and anxiety in both primary and secondary school teachers in comparison with the general population. 109,110 A recent UK study [Wellbeing In Secondary Education (WISE)] of 555 secondary school teachers detected high levels of moderate to severe depression (19%), which were associated with being female, dissatisfaction with work, presenteeism, sickness absence, interpersonal difficulties and low pupil attendance. 110

Alongside lost productivity caused by disability, sickness and presenteeism, the staffing of schools is also challenged by the premature loss of experienced teachers from the workforce; mental ill health is the leading cause of exit from the teaching profession. For instance, it was reported as the main factor in the decision to leave of 46% and 37% of Irish and Scottish teachers. 111,112

Poor teacher–pupil relationships predict subsequent poor child mental health113 as well as poor academic attainment. 18,114 Stressed teachers have more negative classroom management styles and when supported with behaviour management they report reduced emotional difficulties and decreased disruptive behaviour among their pupils. 115 Teachers’ mental health has also been shown to influence the academic attainment of the children they teach. McLean and Connor116 reported that teachers’ depressive symptoms in one term predicted their students’ mathematical abilities a term later. Teachers with less-developed classroom management skills have higher reported overall levels of child classroom aggression, peer rejection and exclusion. 117 In contrast, teachers with highly developed classroom management skills may obtain better results both academically and socially and may reverse the ‘burnout cascade’ to the benefit of themselves and their pupils. 30 There is also an increasing expectation on teachers to identify and manage pupils with mental health needs, which teachers often feel inadequately prepared to do, with consequent effects on teachers’ own well-being. 118,119 Therefore, an intervention that supports teachers to manage disruptive behaviour and promote socioemotional competence has the potential to benefit not only the children they teach but also the teachers themselves.

Previous research has suggested that TCM training improves teachers’ perception of their self-efficacy in relation to their classroom management and that such training has the potential to improve well-being and decrease burnout,94 but the study lacked the power to detect these changes reliably.

Aims

This chapter examines whether or not TCM training leads to a reduction in teacher-reported ‘burnout’ and improved self-efficacy and well-being.

Methods

As with the main trial, this study is reported in accordance with CONSORT and TIDieR guidelines. 120–122 The study design and procedures are presented in full in the published trial protocol,57 which was approved by the TSC and DMC. Ethics approval for the conduct of the trial was obtained from the Peninsula College of Medicine and Dentistry Research Ethics Committee (reference number 12/03/141). The class teachers who were invited to attend TCM training were involved in this substudy, and full details about participant identification, consent, masking and randomisation are presented in Chapter 3.

Results

We recruited a total of 80 trial teachers; 40 were allocated to attend a TCM training programme and 40 acted as controls (Figure 22). Baseline data were collected for all 80 teachers and characteristics were generally balanced between the two arms (Table 25). Follow-up data on the EFQ, TSES and MBI were collected for 74 teachers; three TAU and three intervention teachers did not provide these data. TCM training was well attended, with 36 (90%) of the 40 teachers in the intervention arm attending four or more TCM sessions; 23 teachers (58%) attended all six sessions.

FIGURE 22.

Teachers’ substudy CONSORT diagram.

Table 26 summarises the comparison between the trial arms at follow-up for the EFQ, TSES and MBI measures. There was little evidence of an intervention effect on any of these outcomes, with self-efficacy, burnout and well-being scores changing very little throughout the year.

Discussion

We found no evidence that TCM training reduced teacher-reported ‘burnout’ or that it improved self-efficacy and well-being. These results are surprising in the context of the finding of Marlow et al. 94 that there was an improvement in teachers’ self-efficacy and are in contrast to the process evaluation findings reported in Chapter 6. During FGs and interviews, teachers reported that the TCM training had had a range of impacts on their practice, including changing their mindset, helping them see things from the child’s point of view, helping them develop better relationships with the children, making them think more before responding, helping them feel more positive, more confident and more in control, and helping them create a more positive cycle of behaviour through modelling.

Given these reported changes and past work that demonstrates that behaviour management training reduces teacher-reported stress and emotional difficulties,115 we would have expected to see an increase in teacher well-being and a reduction in their reported levels of workplace stress. A possible explanation of why we did not see these changes could be that we were looking to detect a change in the wrong constructs and that TCM’s impact is being felt elsewhere by teachers. Teachers voluntarily identified with this possibility and throughout the trial made frequent comments to research staff that, although they were able to identify positive changes as a result of the training, the wider context of their professional life was having a greater impact on their well-being, self-efficacy and burnout. As discussed in Chapter 6, teachers report that the biggest impact the training had was on their relationship with the children, but this was not something that we systematically studied.

This substudy was not powered to detect an effect on teachers, so it is still a possibility that a larger sample may reveal intervention effects, but considering our process evaluation the most likely explanation is that we were looking for impacts in the wrong constructs and instead should have been exploring teachers’ relationships with the children they teach.

Introduction

During the FGs carried out as part of our process evaluation, teachers repeatedly reported that although the course had been immediately useful to them, they expected that it would have even more of an impact the following year because they would be able to incorporate all the strategies into their planning and fully embed TCM into their practice. This raised a methodological question for us because in the main trial we followed the children into subsequent academic years because we wanted to include as long a follow-up of the children’s progress as possible. These comments suggested to us that the children with whom the teachers work in the academic year following their training might actually derive more benefit than the intervention children in the main trial who were taught in the same academic year in which the teacher was trained. The effects of the course might also be increased because the children taught in the following year will be exposed to the intervention from the outset of the academic year as opposed to our trial children, whose teachers did not begin the intervention until November and completed it only 1 month or so before our first follow-up measures were collected.

Aims

In this substudy, we aimed to compare the outcomes of children taught by teachers trained in TCM in the previous year (‘Post-TCM’) with those of children in the intervention arm of the trial who were taught by recently trained teachers (‘Concurrent-TCM’). The former group of children, separate from those children in the main trial, were recruited using a convenience sample of TAU schools that accessed TCM training in the final year of the study. Our hypothesis was that these Post-TCM children might derive greater benefit as the teachers would have had more time to embed what they learnt into their practice and to use it in their planning for the academic year.

Methods

Our final round of TCM training was held in the 2015–16 academic year and involved 21 teachers, all of whom were from TAU schools. All children taught by these TCM-trained teachers in the 2016–17 academic year were eligible for inclusion provided that the class teacher judged that they and their parents had sufficient English-language comprehension to understand recruitment information and complete outcome measures.

Written consent was obtained from the headteacher for the school’s participation and from the class teacher for their involvement. Parent information leaflets were sent home with children, and parents were given 2 weeks to ‘opt out’ their child from the research. In order to opt out their child, parents could return a pre-prepared letter or contact the research team directly. Verbal assent was obtained from children each time they were asked to complete a questionnaire.

Outcomes

We replicated the outcome measures and time points of each school’s first year of participation in the main trial, in that all measures were collected during the first academic term of 2016 and 9 months later (T1) in June 2017. Full details of the measures are provided in Chapter 2 and summarised in Figure 23.

FIGURE 23.

Schematic detailing the timing of outcome measures for the Post-TCM substudy.

Results

Fourteen of the 21 schools whose teachers attended TCM training in 2015–16 consented to be part of this study; six schools said that they did not have the time to be involved in this additional work and in one school the teacher who had accessed TCM training had left the school. A total of 397 children were eligible for inclusion in this study; 13 were opted out by a parent, which left a sample size of 384 children. During the year of this additional substudy, two more teachers left the study schools, so follow-up data were not collected for the children in these classes. Our comparison arm (Concurrent-TCM) comprised the main trial’s intervention arm, and full details of this sample are provided in Chapters 2 and 3. Figure 24 depicts the flow of participants through the study and Table 27 compares the baseline characteristics of both arms of the trial.

FIGURE 24.

Post-TCM substudy flow of participants. K, number of schools (clusters); LTFU, lost to follow-up; n, number of children.

Table 28 summarises the comparison between the Post-TCM and Concurrent-TCM groups at the 9-month follow-up. There was little evidence that any outcomes differed depending on when their teacher had completed training, with the exception of a reduction in the percentage of children who scored below national expectations in numeracy for the Post-TCM children (23%) versus the Concurrent-TCM children (33%) (OR 0.45, 95% CI 0.23 to 0.89; p = 0.02).

Discussion

We found no evidence that the impact of TCM training improved children’s outcomes for the children taught by a Post-TCM teacher compared with those children whose teacher was currently undertaking the training, except for numerical attainment, which was better for the children being taught by a Post-TCM teacher. However, attainment data in the main study were measured while the KS 1 and KS 2 curricula were undergoing a number of substantial changes,123,124 and this substudy took place once these changes had been embedded into practice. It is therefore possible that the improvement seen is the result of improvements across academic cohorts and not directly related to changes in the teachers’ schemes of work and classroom practice as a result of TCM training. In addition, given the number of statistical tests undertaken, this finding may just be a chance finding rather than a true difference.

This substudy was limited by the sample of trained teachers and willing schools available to explore the often stated belief that the TCM training would be more effective when the core principles could be embedded in planning. We therefore had limited power to detect differences. We were disappointed not to have been able to recruit more of our TAU schools to this exploratory study, but we are aware that access to training was a major motivator for schools to become involved. This substudy involved yet more disruption to classes after 3 years and four data-collection points, and occurred at a time of pressure due to curriculum change.

Our findings do not suggest that having the full range of TCM strategies available to inform planning from the beginning of the academic year produces additional benefits to the emotional well-being and behaviour of the class. Only a single teacher was trained in each school. Without the ongoing support of a trained peer group, and particularly given the large changes in policy and curriculum during this study, teachers may not have been able to implement the strategies learnt as effectively as they initially thought. It is possible that with ongoing coaching and/or a whole-school approach, teachers’ beliefs about the impact of the course on planning might have been borne out. Further study is needed to make a firm conclusion.

Effectiveness and cost-effectiveness as a universal child mental health intervention

We detected a small but statistically significant improvement in teacher-reported children’s mental health at 9 months (primary outcome: SDQ-TD). Almost all of the 95% CI for the mean difference, however, lies below the assumed minimum clinically important difference (effect size of 0.3 or raw difference of 1.8). Similar effect sizes have been reported on the same measure before and after attendance at CAMHS125 and small effects from universal interventions are common, including the Campbell Collaborations systematic review of TCM strategies. 14,126 The findings provide tentative evidence that TCM may provide teachers with strategies that affect children’s mental health and, although small, the effect detected suggests that this intervention could usefully be tested further.

Small effect sizes are not uncommon in prevention studies. The population-level influence of universal interventions will be differentiated across subgroups, so that the intervention promotes health for those who are currently thriving while preventing deterioration or actively treating others who are vulnerable or struggling. Small population effects, therefore, do not necessarily demonstrate a lack of effectiveness for all subgroups. 14 Indeed, our exploratory but planned subgroup analyses suggest that children with poorer mental health derived the most benefit according to teacher report. This is interesting as it echoes the findings from the trial of TCM and child programmes among high-risk children and classrooms, which also suggested that the intervention led to larger improvements among the children with more severe problems. 30 Similarly, a recent meta-analysis based on study-level summary data of three TCM trials29 also reported a significantly larger effect on children with worse behaviour, and a similar pattern was reported in a meta-analysis based on individual participants of trials of the parallel IY parent training intervention. 6 That the STARS trial has produced a similar pattern of effects adds weight to the argument that the small but statistically significant improvement is more than a chance finding, and suggests that TCM is worth further investigation as a universal intervention in the UK context. Finally, a UK-based cluster RCT of the Good Behaviour Game in 77 primary schools also suggested that boys’ risk of developing conduct disorder demonstrated the most improvement, although the intervention demonstrated little effect on academic attainment or behaviour overall. 127

Given the quantitative findings that those in poor initial mental health seem to respond the most, the comments from teachers in the FGs were more mixed in terms of the impact on children who were struggling, although some were describing children with extremely complex needs who ultimately were not able to cope with mainstream educational provision. These comments highlight the tension commonly described by teachers between supporting children with SEN and undertaking their duty to the rest of the class,128 and it is important to manage the expectations of universal interventions accordingly.

The effect of TCM on the primary outcome was not maintained at 18 and 30 months, which could mean that TCM has no longer-term impact, or could be a result of the children’s reactions to the teaching style of their subsequent teachers who had not accessed TCM training. Given the current financial climate, this intervention was stripped back to the bare minimum for the STARS trial; just one teacher per school was trained. The IY institute currently recommends that teachers are supported during the training by additional coaching,129 and this was an added component of Reinke’s study. 40 Indeed, the consensus from the process evaluation was that a lack of the ‘whole-school’ approach or lack of understanding or support from colleagues interacting with the trial children undermined the effectiveness of the TCM strategies in the classroom. Similarly, the group leaders thought that individual coaching around particularly challenging situations would be a beneficial addition. Most effective universal programmes report that they employ a whole-school approach,14 and the findings from this study would suggest that training all school staff, including learning support and playground assistants, to use the same strategies might amplify and sustain any initial impact on children’s mental health that training a single teacher might have.

The teachers and group leaders in the process evaluation were arguing for the use of the IY TCM approach throughout the school and the training of a critical mass (if not all) of staff when they used the term ‘whole-school approach’. Others would reserve this term for broader programmes that worked at universal, targeted and indicated levels of prevention simultaneously, which often include a mixture of environment, didactic and individualised content. 130,131 The training of a critical mass of school staff in IY TCM shows promise. A quasi-experimental Norwegian evaluation matched 21 schools in which all Grade 1–3 teachers (children aged 6–8 years) were trained with 22 matched schools that did not receive training. 132 The comparison of pre and post scores between children in TCM and TAU schools suggested a small improvement in social skills and a very small improvement in behaviour. This study involved the simultaneous training of staff with pre and post measures collected in the same academic year. The teachers in the STARS trial believed that any impact would increase in the following year when they could apply their strategies to planning as well as delivery, and children in the Norwegian study were not exposed to the IY strategies for very long. The effects of a whole-school approach may, therefore, increase over time. A critical mass of staff might also provide longer-term support for each other after the training was completed, which might also enhance any initial impact TCM had, as might the continued provision of coaching after TCM. Future research should test both the coaching and the training of more school staff in the UK context.

The addition of other IY programmes, which were initially conceptualised as a multistrategy approach to children with severe behavioural difficulties31 might further enhance the impact of IY TCM. In southern Ireland, such a combined programme demonstrated elevated SDQ scores at school entry and reduced subsequent SDQ scores in a non-randomised evaluation of schools that implemented IY TCM compared with those that did not. TCM was combined with access to the IY parenting programmes and the use of the IY classroom Dina programme for children. 133 This report, plus those reporting on other whole-school programmes, provide some important lessons for implementation. 130,131,133,134 As highlighted by the Gatehouse project (an Australian, multilevel, integrated whole-school approach to well-being in secondary schools),130 a clear and broad conceptual framework, plus support for schools to respond to their own context and own data, appears key to successful implementation, along with time and resources for reflective practice and sustainability. A recent US-based cluster RCT of the School-Wide Positive Behaviour Intervention and Support in Elementary schools, which was followed by a related implementation study, demonstrated improved behaviour, punctuality and attendance plus reduced exclusion. 135,136 The theoretical basis of this programme is very similar to that of IY TCM, with an emphasis on proactive behaviour management and non-punitive responses. Interestingly, effectiveness was correlated with the extent to which schools implemented the different elements of this whole-school approach.

There may be a methodological explanation for the lack of effect at later follow-up. Baseline mental health measures and follow-up after 9 months were completed by the same teacher, whereas these measures were completed by two different teachers at 18 and 30 months. As would be expected, the ICCs were markedly larger for teacher-reported SDQ-TD score (0.12–0.18) than for the corresponding parent-reported score (0.06), which suggests that variability in how teachers score their pupils may contribute noise at the two later follow-ups.

Furthermore, children in the intervention arm were exposed to TCM strategies for a relatively short duration. In the process evaluation, teachers repeatedly discussed their belief that the TCM training would have a greater effect in the subsequent academic year because they could embed the strategy into their planning as well as their teaching, and the children would be exposed to these strategies from the outset of the academic year. This led us to collect data from the Post-TCM comparison study, which did not provide empirical support for their belief. Recruitment to this substudy was disappointing, with only 67% of eligible schools consenting and a further 10% unable to provide follow-up data owing to the teacher leaving the school. This limited the power available to detect a difference between the Post-TCM sample and the Concurrent-TCM sample as the large CIs (OR 0.72, 95% CI –0.48 to 1.92) demonstrate that it is plausible that there are benefits in either direction. This additional work used a convenience sample of TAU arm schools that had already participated for 3 years in the STARS trial and occurred in an academic year when primary schools were having to grapple with substantial changes in the primary school curriculum. 123,124 That participation was lower than anticipated is therefore, perhaps, not surprising. This abrupt change in the curriculum and its impact on the school context, teachers and children may have introduced time or cohort effects that we were unable to adjust for in our analysis. We lacked any process evaluation in this small additional study that might have offered alternative explanations or insight into these processes. The Post-TCM study is sadly inconclusive but certainly does not provide evidence that future studies should follow up the teachers’ subsequent classes rather than the children they were teaching as they accessed the course, which is the design chosen by all other trials to date. As stated above, it is possible that a whole-school application of TCM may amplify any effect of TCM when one teacher per school is trained, as children could then benefit from the intervention throughout all their remaining school years.

Although TAU children were never taught by a TCM-trained teacher, in the majority of TAU schools there was a TCM-trained teacher at their school during their second and third years of the study. It is therefore possible that TAU children were exposed to elements of TCM through contamination effects, which would reduce the difference between trial and intervention arms. The fact that all schools eventually accessed TCM will, however, have concealed allocation in many schools; interviews with SEN co-ordinators, whose roles relate to behaviour management, revealed that they were often unaware of the school’s involvement in the study. 137 Ideally, future research would not provide delayed training to TAU schools but rather offer some other type of incentive for participation.

The small but sustained effects on disruptive behaviour and inattention/hyperactivity as measured by the teacher SDQ across all three follow-ups are interesting and warrant replication. More than 40% of secondary school children reported in a survey that their classroom was too noisy to work in,138 and overactivity/inattentive traits predict poorer academic attainment at General Certificate of Secondary Education (GCSE) level. 139 Children in poorly managed classrooms observe that disruptive behaviour commands staff attention, which may then increase later disruptive behaviour as well as disengagement from school, with its attendant risks to health and education among quieter, less disruptive pupils. The potentially positive impacts of TCM are, therefore, exciting and should be explored further in future research. The linkage to the NPD demonstrated some interesting differences in relation to exclusion between the trial arms, which suggest that these data may provide a useful vehicle through which to conduct longer-term follow-up of the STARS participants. However, no firm conclusions can be drawn from the NPD data because only a very small number of children were excluded in the STARS trial, which is what would be expected in primary schools.

We failed to detect any influence of TCM on the HIFAMS measure. Some researchers would recommend 8 years of age as the minimum140 for reliable reporting on standardised measures of mental health, although increasingly researchers are seeking reports from younger children. HIFAMS was developed for the STARS trial and has demonstrated validity and moderate reliability among children as young as 4 years when tested in this and two other samples. 52 It has now been translated into Italian and Persian and is being used in several education-based studies in the UK. Our findings would not suggest that being taught by a TCM-trained teacher leaves children happier about their school, but, equally, the study was not powered to detect an effect using this measure. It would seem important to try to capture children’s experiences directly, because the correlation between parents and children is known to be low. 141

We also did not detect an intervention effect on parent-reported measures, but this was not unexpected. Children respond differently in different situations and TCM targets the classroom rather than the home. In addition, a low level of agreement between parents and teachers is common in child mental health research,141 so the lack of effect on parent-reported outcomes does not necessarily indicate a lack of effectiveness in the school context. Parents will have only second-hand knowledge of how their children function in the classroom after all.

In terms of cost-effectiveness, the primary economic evaluation, using SDQ-TD scores, suggested that TCM may be cost-effective compared with TAU at 30 months, particularly at higher levels of willingness to pay for improvements in SDQ-TD scores. These results were not altered in a sensitivity analysis controlling for additional randomisation variables (CC2) but were affected by the impact of missing data (MI), which showed that the probability of TCM being cost-effective was reduced at lower willingness-to-pay thresholds for a unit improvement in total SDQ-TD scores. Although the SDQ is a commonly used measure of mental health in children and adolescents, it is not associated with a willingness-to-pay threshold for decision-making and is therefore of limited use in allocating scarce health-care resources. With very small differences in costs and effects, as in this study, it is not possible to draw firm conclusions about the cost-effectiveness of TCM without knowing society’s willingness to pay for improvements in SDQ-TD scores.

Cost–utility analyses using QALYs generated using a mapping algorithm suggest that TCM has a higher probability of being cost-effective than TAU at 9- and 18-months but not at 30-months follow-up, which is consistent with the clinical findings that suggest that early positive findings were not maintained over the longer term.

Small differences in SDQ-TD and mapped utility scores over the trial follow-up made extrapolation beyond the trial period unnecessary, as the baseline beyond which data would be extrapolated from the end of the trial period was almost identical in the two groups. Longer-term analysis would therefore serve only to replicate the within-trial results. Although small clinical differences can result in large changes in health outcomes over longer-term models, there was limited evidence to suggest this in the trial data, particularly given the sharp drop in the probability of TCM being cost-effective using QALYs from approximately 80% at 18 months to 5% at 30 months. It is therefore fair to assume that any projections of costs and effects beyond 30 months would be of questionable validity. Appropriate temporal extrapolation of costs and utilities is reliant not only on clinical outcomes but also on many other long-term processes such as adaptation to particular health states, data on which are not currently available for children and adolescents.

Fidelity of delivery

The reported checklists completed by the group leaders and teachers, along with the monthly supervision from the programme developer, demonstrate that the course was delivered with fidelity to the model. In addition, we demonstrated, in those schools where T-POT observations were carried out, that the training changed the behaviour of teachers back in the classroom. Teachers in the intervention arm employed more praise and, paradoxically, more indirect commands, and child compliance increased. The two other studies that have used the T-POT observational measure38,39 both detected a decrease in negative strategies rather than an increase in positive strategies, but Hutchings et al. 38 focused their observation on key high-risk target children, rather than the whole class, so the findings are not directly comparable. The observed increase in the use of praise mirrors the findings from observations of parent–child dyads in trials of the IY parenting programmes, but we did not detect any reduction in the use of negative strategies that we would have expected. 142 We were resourced only to conduct observations in one-quarter of participating schools, which limited our power to detect changes in the strategies employed, and may have introduced biases if there were systematic differences between participating schools and those who chose not to participate or were not invited to do so. The observers were blind to treatment allocation and the schools were selected on the basis of cohort (cohort 1 or 2) and geography (cohort 2 only; schools in and around Exeter) rather than enthusiasm (or lack thereof) for the course. The T-POT measure was chosen for comparability with other European trials of IY. Future research should consider observations in a larger proportion of classes, if not all of them. Given the issues that have arisen in previous studies that have focused on target children when those children are absent or have little interaction with the teacher during the observation period, we would recommend observations that focus on the whole class. A review of moderators of the parallel parenting intervention suggests that observations may be more sensitive than questionnaire measures. 143 However, they are time and resource intensive in terms of ensuring that researchers are sufficiently trained to conduct and record the observation reliably and in organising when they can slot into the school timetable without disruption that might alter teacher behaviour and thus invalidate the observation. As soon as a teacher alters their behaviour to accommodate the requirements of the observation, then, arguably, the observation is unable to be considered valid and the results should not be regarded as any more reliable than questionnaire measures.

Academic attainment

In the process evaluation FGs, teachers reported that children were calmer, more motivated and more ready to learn, as our logic model predicted. These reported experiences did not translate into empirical evidence of an improvement in academic attainment at a universal level. The interaction with baseline mental health, which is difficult to interpret in terms of direction, is interesting and suggests that the impact on attainment is worth studying in more depth. It is intuitively plausible that a more settled classroom would allow accelerated progress for children who were otherwise thriving. Given the crudeness of the measure available to us, we were not surprised that there was no main effect detected on attainment. It would not have been feasible to conduct individual-level attainment tests at four data points for > 2000 children, and others have commented how difficult it is to find appropriate measures of attainment for use in education-based trials. 144 The linkage to the NPD will allow us to explore whether or not the improved attention suggested by the SDQ Hyperactivity scale translates into better academic attainment over time when all these pupils have taken their national exams in Years 6 (2022) and 11 (2027). A population-based study has suggested that each 1-point increase in teacher reported inattention/hyperactivity symptoms at 7 years of age was associated with a 2- to 3-point decrease in GCSE score and a 10–12% increase in the likelihood that the pupil would not obtain five GCSEs at grade C or above, including in maths and English. 145 Arguably, evidence of ‘real world’ impacts on attendance, exclusion, SEN and qualifications are more useful and powerful measures than standardised tests, although these children will have been exposed to a great deal of other influences during their school careers. The interesting findings from the Head Start Programme,146 which began in the USA in the mid-1960s, suggest that even a brief pre-school intervention may produce impacts more than a decade later and suggest that these longer-term influences might be present in the STARS sample.

Teacher mental health, burnout and self-efficacy

In the FGs, teachers reported that attendance at the course changed their mindset, increased their focus on children’s needs, improved their relationships with pupils and parents, helped them to be more proactive and less reactive, and left them feeling calmer, more confident and more in control. These findings were predicted by our logic model (see Chapter 1) and suggest that TCM was experienced as helpful despite teachers’ many competing priorities. There is a stark contrast between the highly positive endorsement of the course by these teachers and group leaders and the lack of impact on teacher’s mental health, self-efficacy and burnout in the quantitative measures.

Teachers’ comments during the process evaluation and anecdotally during the course of data collection (see Chapter 6) suggest that although the TCM training was hugely helpful to them in the classroom, the external and internal pressures on primary schools have escalated during the course of the study. Teachers, particularly in the current climate of reduced external support and shrinking budgets, are juggling many competing priorities. Given the impact of poor mental health on academic outcomes,16,17 and the regular and longitudinal contact with children and their families, schools are necessarily a key forum for mental health promotion and prevention, now enshrined in English government policy. 119 Our findings suggest that IY TCM was perceived to be useful and relevant to the practice of participating teachers.

Many teachers commented on the pressures they face, which may contribute to the shockingly high and sustained levels of psychological distress that we detected among teachers in this study; 10% scored above a cut-off point that suggests moderate depression at all four data points. 147 These high levels of stress and distress in the wider school environment may also have served to undermine any positive influence of TCM on self-efficacy, particularly in relation to the classroom management subscale, on which we did detect a statistically significant improvement in our uncontrolled feasibility work. 94 It may also be that our change to the short Teacher Self-Efficacy Scale in an effort to reduce participant burden rendered the measure insensitive, so future studies should consider using the original longer form148 as we did in our feasibility work. In addition, some teachers reported that what changed most in the classroom was their relationship with the children, so future studies should consider including a measure of the teacher–pupil relationship. The impact of teacher–pupil relationships should not be underestimated; poor teacher–pupil relationships predict poor mental health, particularly behaviour problems, and were also related to poor family function (primary school-age children only) and future exclusion (secondary school-age children only) in a large population-based study of primary school children. 113

The feasibility and acceptability of the Teacher Classroom Management course

The high levels of attendance and positive feedback from teachers and group leaders in our process evaluation suggest that TCM is an acceptable and feasible course for primary school teachers in the UK context. Two-thirds of teachers attended all sessions offered, and 89% attended four or more sessions. Other evaluations of this programme have achieved higher attendance by offering ‘catch up’ sessions to teachers who had to miss a session, which the STARS trial lacked the time and resources to do. 30 However, the CACE analysis suggests that increased attendance would not have increased the effectiveness of the intervention substantially, so we lack evidence to endorse the provision of catch up sessions.

The teachers reported that they valued the structure of the course, the opportunity to ‘see, practice and implement’ TCM strategies and the chance to discuss their experiences. The collaborative ethos of the course and the time to share and reflect outside their classrooms were also highly valued. On the whole, teachers rated aspects of the course, such as the course content, video clips, discussion and delivery of the course, as either ‘helpful’ or ‘very helpful’, although, in general, ‘satisfaction’ type questions are often answered very positively after courses and workshops in all fields (see Chapter 6).

Teacher-reported facilitators to learning included an understanding of the theory behind the strategies, as endorsed by the TCM training, the adaptability and flexibility of the TCM strategies and the fact that the strategies were generally easy to use. Teachers mentioned that they used a number of different strategies as a result of their attendance at the TCM course, which included specific and proximal praise, ignoring, rewards and consequences, time-out, emotional regulation and behaviour plans. They also reported attempting to use coaching and commentary, although the feedback on this was more mixed, which may indicate a technique less familiar or less appropriate to the UK context, or one that group leaders need to increase the focus on. Limitations to implementation were mainly external to the course, as reported by all stakeholders in our process evaluation, and included a lack of support from colleagues or senior management teams, clashes with school behavioural policy, and fears about the perception of TCM strategies by external bodies such as Ofsted. However, these barriers are potentially tractable if senior managers and school inspectors are brought on board to ensure effective implementation within both schools and teacher training models.

Methodological issues

The internal validity of the STARS trial derives from the high retention of schools, teachers and pupils over 30 months, the delivery of TCM with fidelity by experienced practitioners trained and supervised by the programme developer, independent randomisation and the use of a strongly validated and widely used primary outcome measure. Trial arms were balanced in terms of school characteristics. There were small imbalances in terms of children (more poor readers and 4-year-olds in the intervention arm) and teachers (higher levels of psychological distress, lower self-efficacy, fewer qualified for > 5 years and older in the intervention arm), which may have made it harder to detect an effect on teacher outcomes and academic attainment.

High levels of attendance suggest that teachers valued TCM, and the participating schools were generalisable to the UK population in terms of class size and eligibility for free school meals. The trial benefited from a rigorous cost-effectiveness analysis and a detailed process evaluation.

Like all studies, the STARS trial had some methodological limitations. It was not possible to mask teachers to their allocation, risking response bias, particularly for the 9-month follow-up when intervention teachers reported on behaviour and mental health outcomes following attendance on a course designed to specifically improve these. Thus, our small positive effect at 9 months may be either a chance finding or may be due to teachers’ enjoyment of the course and their belief that it was effective, rather than a true indication of effectiveness. However, outcomes at 18 and 30 months were completed by different teachers who did not attend the training and the decrease in both the teacher-reported SDQ Hyperactivity subscale and the PBQ across all follow-ups undermines the argument that the findings can be wholly explained by reporting bias, despite the negative findings on the primary outcome at this time point. Moreover, the blind observations that we were able to undertake did detect changes in teacher behaviour and in child compliance on some strategies that were similar to the findings of previous studies in this area. 38,39

Similarly, the size of the research team and the need to develop close and sustained relationships with such a large number of schools, which underpinned our success in recruitment and retention, also meant that only the researchers who conducted observations within the classroom were blind to allocation status. Most measures were completed independently; teachers completed their measures directly on to a database, and parents completed theirs via mail-outs in the book bag or direct to their home. Researchers supported children under the age of 7 years to complete HIFAMS individually, whereas children in KS 2 completed this questionnaire as a whole-class activity, which allows little opportunity for the researchers to influence responses. In addition, we did not detect a difference by trial arm on this measure.

The exclusion of schools that lacked a substantive headteacher or were judged as failing in their last Ofsted inspection was adopted to protect the internal validity of the study. The burden of research was seen as inappropriate for struggling schools by our advisory group, who also reported that many schools in this situation change senior management team, with a high likelihood that the incoming leads might withdraw their school. This is in fact what occurred with the only school that was lost. In addition, such schools would already have had the involvement of a number of external organisations that would be requiring changes and we wanted to evaluate TCM without these additional influences. This choice reduces the generalisability of STARS, as it is highly likely that such schools experience greater difficulty in managing classroom behaviour. The distribution of teacher SDQ-TD scores at baseline, however, was almost identical to that from population surveys of children of this age, which suggests that any selection bias was minimal. 49

Headteachers are used to considerable autonomy, and it was clear from our feasibility work that any attempt to control the selection of teachers would be a major disincentive to their school’s participation in the study. There are two potential biases that might occur from headteachers’ selection of teachers to attend the course. If teachers were selected because they struggled with behaviour management, we might overestimate the impact of the intervention, whereas if they were selected because of a particular interest in socioemotional well-being, we might underestimate the impact if interest correlates with skills, or overestimate the impact if interest correlates with receptiveness. As the selection of teachers preceded randomisation, it should not have compromised the internal validity of the study and reasonable balance was obtained on teacher characteristics (see Table 4). Our process evaluation involved interviews with headteachers and suggested a number of reasons for their choice of teacher to nominate, which included NQT status, allocation of a class known to be particularly challenging or a known interest in behaviour management.

Some might query why we detected a difference on the PBQ across all three follow-ups according to teachers but no difference on the Conduct Subscale of the SDQ according to teachers or parents. The former was derived from the longer scale used in the School Effectiveness Literature149 and so focused on common low-level disruptive behaviours that impede effective teaching and are the direct targets for TCM. It covers talking out of turn, interrupting other pupils, unnecessary non-verbal noisiness, cheeky or rude comments to the teacher, and verbal or physical aggression towards peers. In contrast, the SDQ Conduct Subscale focuses on more antisocial behaviours, such as cheating, lying, stealing, tantrums and disobedience, which are uncommon in this age group in a mainstream school context, are less easily influenced by teachers’ general classroom management strategies and are not so closely aligned to the aims of the TCM training. We did not anticipate changes on this subscale, which was the reason for the inclusion of the PBQ.

Finally, there were some potentially interesting background characteristics of the children about whom we lacked information, such as a previous history of poor mental health either in themselves or in their family. There is always a tension between what it is desirable to know about and the risk of overburdening participants, which risks incomplete data.

The contrast between the highly positive reports from teachers in the process evaluation and the tentative evidence of effects from the quantitative findings is interesting and puzzling. FGs set out to invite all teachers who attended the TCM training (although data collection stopped when data saturation was reached); response rates were high and teachers were incentivised to attend, and we also actively encouraged those who could not attend and who had dropped out of the course to discuss the same topics via an interview. It seems unlikely, therefore, that these teachers were particularly enthusiastic and they did not seem to differ systematically from the rest of the sample of participating teachers. Teachers who took part had varying levels of experience, were of varying ages and worked within both KS 1 and KS 2. The FGs and interviews themselves were conducted using a robust, reproducible methodology and a published protocol. 101

However, like the main trial, the process evaluation also has limitations. Although every effort was made to ensure that the FGs and interviews were conducted by a researcher who had not acted as the main point of contact for the teacher during their involvement, researchers were not independent of the STARS trial and may have been known to the teachers. This could have led to information bias if teachers felt inhibited in reporting negative results. 150 Given that feedback was consistent across FGs and that interviews were conducted with > 40 teachers, we are confident that we minimised such potential.

Although we successfully obtained data from > 70% of parents at baseline, attrition was marked. Response rates for the CA-SUS were particularly affected by low parent report, with only 48% responding. Although MI of the missing data did not markedly affect the results, any conclusions reached must be considered alongside this low response fraction.

The economic evaluation was limited by the need for mapping to create QALYs. Although NICE recommends the use of mapped health state utility estimates when directly collected data are not available,73 the validity of these mapping methods has not been fully addressed. Mapping can result in loss of information and increased uncertainty and thus requires a closer assessment of the agreement between target and source instruments. It has been argued that current methods for mapping are not known to be conceptually robust88 and carry a ‘significant risk that may be harmful to population health’. 89 It is therefore important to consider results of the mapped cost–utility analysis with greater uncertainty. It is possible that vital information is lost when mapping five subscores of the SDQ (emotional symptoms, conduct problems, inattention/hyperactivity, peer problems and a pro-social scale) to the nine dimensions of the CHU-9D (sad, worried, pain, annoyed, tired, homework or schoolwork, daily routine, activities and sleep), which could therefore lead to erroneous conclusions about cost-effectiveness. Authors of the mapping algorithm used in the present study noted that, although the algorithm performed well in predicting mean group observed utility values, it did not perform well in predicting at individual participant level. 78 Despite published guidelines and a checklist for the reporting of mapping algorithms,90,91 only a few studies have been found to fulfil or partly fulfil all criteria on the checklist. 92

It was not possible to calculate QALYs because the CHU-9D is not suitable for the very young children included in this sample, whose ages ranged from 4 to 9 years at recruitment and 6 to 11 years at final follow-up. 151 Feasibility work that we conducted of a child version of the EQ-5D suggested that our very young sample struggled to understand the concepts involved and that the measure was not valid or reliable. 76

The economic evaluation was also limited by failing to consider spillover effects in teachers’ quality of life. TCM has been designed to increase the well-being of teachers as well as of students, and future research should attempt to capture these effects by directly administering a preference-based quality-of-life instrument such as the EQ-5D, which is commonly used in adults. 93 However, the results presented in Chapter 7 suggest that there was no evidence that TCM training improved teacher well-being, so the impact of failing to measure teacher utility is likely to be limited.

Future research

The findings of the STARS trial emphasise the importance of the school context to children’s mental health and provide tentative evidence that teacher training and continuing professional development might usefully explore how classroom management style may influence children’s mental health, behaviour and attainment. Future studies should consider the following questions:

1. How does TCM influence teacher–pupil relationships?

2. If children are taught by teachers who have accessed TCM in 2 (or more) sequential years, is the impact on the mental health of children sustained or even amplified?

3. Does the impact of TCM increase in the year after the teacher has accessed the course as teachers reported it would?

4. What is the influence of TCM training on robust measures academic attainment in the short and long term?

5. Does the addition of the IY parenting course and/or the IY Dina programme for children with particularly challenging behaviour enhance the effectiveness of TCM? Is this cost-effective?

6. Does the addition of individual coaching for teachers outside the TCM sessions amplify the effectiveness of TCM? Is this cost-effective?

7. Would TCM be effective in schools with particularly high levels of behavioural disturbance, such as Alternative Provision for excluded children?

8. Does the addition of individual coaching for teachers during and after the course increase the effectiveness and cost-effectiveness of TCM?

9. Does the level of behavioural challenge that the school is faced with moderate the impact of the course and the effect of adding coaching or parent training?

There are some methodological issues that could also be explored, such as the extent to which blind observations correlate with questionnaire measures and whether or not this differs by informant. The former are expensive and time-consuming and require considerable training and reliability checks to yield reliable data. Implementation studies could address the extent to which schools that appear to be struggling because they lack a substantive headteacher or were judged as failing by Ofsted were able to implement the research and the intervention. It would be important to identify barriers to and facilitators of including such schools to inform supportive strategies. Both could inform the design of future trials.

There is also scope for some interesting secondary analyses of the existing data, which include the longer-term follow-up of educational outcomes via linkage of the data to the NPD. In addition, we plan to undertake a mixed-methods ‘following the thread’ analysis to explore the relationship between the strategies adopted by teachers from the TCM training, teacher–pupil relationships and child mental health using both qualitative and quantitative data in order to refine the intervention theory to better inform future research.

Conclusions

The findings of the STARS trial provide strong evidence that TCM is feasible and acceptable in the UK context, and, in those schools in which observations were carried out, led to changes in teachers’ behaviour and improved compliance from children in blind observations. There is some evidence that being taught by a teacher who has attended the course may reduce low-level disruption in primary school classrooms and be associated with improved attention and concentration over 30 months. There may also be differential effects according to baseline mental health in terms of the influence of TCM on mental health and academic attainment, but there was no evidence that TCM improved children’s happiness at school or teacher mental health, burnout or self-efficacy.

Although there is insufficient evidence to recommend widespread implementation, these findings suggest that further evaluation is warranted. Future research should explore the impact of training more school staff in TCM and focus on the impact on teacher–pupil relationships and academic attainment in addition to mental health, behaviour and concentration. TCM might generate the most impact if employed as a whole-school approach, with or without the provision of additional individual coaching or parental support. These are questions that should be tested empirically. There are also obvious implementation issues to work through, such as the practicalities for schools of simultaneously releasing multiple teachers for training, and the need to have adequately trained and supervised group leaders to ensure fidelity. One solution would be to provide TCM as part of initial teacher training or within the NQT programme.

Contributions of authors

Tamsin Ford (Professor of Child and Adolescent Psychiatry, Child Health) as chief investigator was responsible for the original proposal, securing funding, writing the statistical analysis plan and the initial draft of this manuscript, and had overall responsibility for the management of the trial.

Rachel Hayes (Research Fellow, Child Health) was responsible for writing the statistical analysis plan, data collection and cleaning, management of the trial and writing the initial draft of this manuscript.

Sarah Byford (Professor of Health Economics) was responsible for the original proposal, securing funding, writing the statistical analysis plan, the management of the trial, co-ordinating the health economics analysis and writing the initial draft of this manuscript.

Vanessa Edwards (Senior Research Fellow, Child Health) was responsible for the original proposal, securing funding, data collection and the management of the trial.

Malcolm Fletcher (IY Group Leader, Behavioural Consultant) was responsible for the original proposal, securing funding, the management of the trial and delivering the TCM training.

Stuart Logan (Cerebra Professor of Paediatric Epidemiology, Child Health) was responsible for the original proposal, securing funding and the management of the trial.

Brahm Norwich (Professor of Educational Psychology and Special Educational Needs) was responsible for the original proposal, securing funding and the management of the trial.

Will Pritchard (IY Group Leader, Behavioural Consultant) was responsible for the original proposal, securing funding, the management of the trial and delivering the TCM training.

Kate Allen (Associate Research Fellow, Child Health) was responsible for data collection and cleaning, and for writing the initial draft of this manuscript.

Matthew Allwood (Associate Research Fellow, Child Health) was responsible for data collection and cleaning.

Poushali Ganguli (Research Associate, Health Economics) was responsible for co-ordinating the health economics analysis and for writing the initial draft of this manuscript.

Katie Grimes (Associate Research Fellow, Child Health) was responsible for data collection.

Lorraine Hansford (Associate Research Fellow, Child Health) was responsible for data collection and cleaning and for writing the initial draft of this manuscript.

Bryony Longdon (Associate Research Fellow, Child Health) was responsible for data collection and cleaning and for writing the initial draft of this manuscript.

Shelley Norman (Associate Research Fellow, Child Health) was responsible for data collection and cleaning.

Anna Price (Associate Research Fellow, Child Health) was responsible for data collection and cleaning.

Abigail Emma Russell (Research Fellow, Child Health) was responsible for conducting the T-POT analysis and for writing the initial draft of this manuscript.

Obioha C Ukoumunne (Associate Professor in Medical Statistics) was responsible for the original proposal, securing funding, writing the statistical analysis plan, the management of the trial, conducting the statistical analysis and writing the initial draft of this manuscript.

All authors contributed to, and approved, the final manuscript.

Publications

Ford T, Edwards V, Sharkey S, Ukoumunne OC, Byford S, Norwich B, Logan S. Supporting teachers and children in schools: the effectiveness and cost-effectiveness of the incredible years teacher classroom management programme in primary school children: a cluster randomised controlled trial, with parallel economic and process evaluations. BMC Public Health 2012;12:719.

Marlow R, Hansford L, Edwards V, Ukoumunne OC, Norman S, Ingarfield S, et al. Teaching classroom management – a potential public health intervention? Health Education 2015;115:230–248. https://doi.org/10.1108/HE-03-2014-0030

Hansford L, Sharkey S, Edwards V, Ukoumunne O, Byford S, Norwich B, et al. Understanding influences on teachers’ uptake and use of behaviour management strategies within the STARS trial: process evaluation protocol for a randomised controlled trial. BMC Public Health 2015;15:119. https://doi.org/10.1186/s12889-015-1486-y

Marlow R, Norwich B, Ukoumunne OC, Hansford L, Sharkey S, Ford T. A comparison of teacher assessment (APP) with standardised tests in primary literacy and numeracy (WIAT-II). Assess Educ 2014;21:412–26.

Nye E, Gardner F, Hansford L, Edwards V, Hayes R, Ford T. Classroom behaviour management strategies in response to problematic behaviours of primary school children with special educational needs: views of special educational needs coordinators. Emot Behav Difficult 2015;21:1–18.

Allen K, Marlow R, Edwards V, Parker C, Rodgers, L, Ukoumunne OC, et al. ‘How I Feel About My School’: The construction and validation of a measure of wellbeing at school for primary school children. Clin Child Psychol Psych 2017;23:25–41. https://doi.org/10.1177/1359104516687612

Price A, Allen K, Ukoumunne OC, Hayes R, Ford T. Examining the psychological and social impact of relative age in primary school children: a cross-sectional survey. Child Care Health Dev 2017;43:891–8. https://doi.org/10.1111/cch.12479

Ford T, Hayes R, Byford S, Edwards V, Fletcher M, Logan S, et al. The effectiveness and cost-effectiveness of the Incredible Years^® Teacher Classroom Management programme in primary school children: results of the STARS cluster randomised controlled trial [published online ahead of print July 18 2018]. Psychol Med 2018.

Allwood M, Allen K, Price A, Hayes R, Edwards V, Ball S, et al. The reliability and validity of the pupil behaviour questionnaire: a child classroom behaviour assessment tool. Emot Behav Difficult 2018;23:361–71.

Data-sharing statement

The corresponding author had full access to all of the data in the study and final responsibility for the decision to submit for publication. Parents were informed that the research data would be accessed only by the research team and, therefore, supporting data cannot be made openly available. All data requests should be submitted to the corresponding author for consideration. Access to anonymised data may be granted following review.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the PHR programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the PHR programme or the Department of Health and Social Care.