Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology

Speed, quality of product and patient outcomes

These three outcomes, while not formalised assessment processes, have been explored for use as surrogate measures of surgical skill.

Operative speed may provide a measure of technical skill, although there is a paucity of literature on this subject. Robert Liston, a London surgeon working before the introduction of anaesthesia, was proud of his operative speed and once challenged observers, ‘Now gentlemen, time me, 28 seconds before placing an amputated limb in the sawdust’. 63 However, measuring competence merely by setting time targets for a certain procedure is crude and probably unacceptable, as a fast surgeon is not necessarily a good one. Lord Lister, in comparison, was observed to have ‘none of the dramatic dash and haste of the surgeon of previous times. He proceeded calmly, deliberately and carefully and, as he told his students, anaesthetics have abolished the need for operative speed and they allow time for careful procedure’. 63

The Toronto group undertook a small study to examine whether time and operative product could serve as measures of technical skill using bench model simulations. 64 Twenty general surgery residents participated in a six-station bench model examination, in which ‘time to completion’ was recorded and ‘quality of the final product’ was assessed using a global five-point rating scale by two assessors per station. The mean inter-rater reliability was 0.59 for product quality. Interstation reliability (Cronbach’s α) was 0.59 for analysis of product quality and 0.72 for time to completion. Both measures demonstrated construct validity, with more senior trainees producing better products in less time, although there was poor agreement with previous OSATS examination scores. These measures offer a time- and cost-efficient, if less reliable, alternative to direct observation for the assessment of technical skill. However, the quality of the final product may be relatively easy to measure on a simulation, whereas it may be more difficult to assess in the operating theatre and is likely to depend on the specific procedure. Similarly, time to completion is a straightforward measure for simulations but in the operating theatre is affected by many case-specific variables as well as the performance of the whole surgical team.

Expert surgeons appreciate that operative speed is important. For instance, surgeons performing cardiac and vascular procedures seek to minimise cardiac bypass and vessel cross-clamping time to reduce operative complications. Some evidence with respect to time and surgical performance comes from the Medical Research Council (MRC) European Carotid Surgery trial, which reported a relationship between procedure time and adverse outcomes. 65

Efficiency in operating time is important to maximise service delivery, which is increasingly pressured under the current policies for patient management pathways and waiting list times. However, such efficiency requires large numbers of procedures, which trainees could not usually expect to obtain, and is therefore a more appropriate aspiration for newly appointed consultants.

Although attractive, measurement of the performance of surgeons based upon patient outcomes is fraught with difficulty owing to variation in case mix and the large numbers required for reliability. 66 Errors made by trainees are often corrected, and therefore masked, by their supervising consultant. In addition, patient outcomes reflect the performance of the whole surgical team, both within the operating theatre and during the postoperative period, and therefore do not provide a reliable assessment of an individual surgeon. It may be a good screening method for consultant surgical skill, but tests of competence will be required for those consultants for whom there is cause for concern.

OpComp

Logbooks form a useful record of procedural experience67 but do not reflect the performance level achieved by trainees, lacking both validity and reliability as a method of assessing trainees’ surgical skill. 68 The Operative Competency (OpComp) form was introduced by the Specialty Advisory Committee (SAC) in General Surgery in 2003 to complement the information provided by logbooks on trainees’ surgical skills. 69 The OpComp form asked educational supervisors to assess the ability of a trainee to perform the specific index procedures (relevant to their specialty) against defined criteria at the end of a clinical placement. The following rating scale was used to make these summary judgements, derived from the then current ‘Training the Trainers’ Course70 and modified on the basis of the pilot studies:69

U = Unknown or insufficient evidence to support a judgement.

D = Unable to perform the procedure, or part observed, under supervision.

C = Able to perform the procedure under supervision.

B = Able to perform the procedure with minimum supervision but needs occasional help.

A = Competent to perform the procedure unsupervised and can deal with most complications.

A checklist of technical skills was provided on the reverse of the OpComp form to assist educational supervisors in making judgements. This checklist’s content was systematically derived using a Delphi survey of surgeons in Scotland to establish the ‘essential’ technical skills required of trainees. 71 The OpComp form was shown to have good construct validity, and trainers found it simple to complete. 68 Although an advance, there were aspects of this method that undermined its reliability. The form suffers from retrospective recall of numerous procedures over the course of a placement, risking loss of important training information and cross-contamination between procedures. Almost half of the trainees surveyed during its pilot indicated that a trainer had rated their ability to perform a procedure unseen. 68 In addition, the reliance on a single assessor per clinical placement opens up this method to various types of assessor bias, including the halo effect, whereby an assessor provides a final opinion rather than providing a discriminatory rating for each item,72 and expectation bias, in which the knowledge of a trainee’s seniority influences assessors’ ratings,73 as well as bias arising from the primary influence of a trainee’s interpersonal skills, rather than their technical skills, on supervisor ratings. 74 The use of a collection of surgical skill assessments at the end of a clinical placement did not promote the opportunity for ‘on-the-job’ training and feedback. It can be appreciated that the move to current WBAs, using immediate assessment and feedback after single procedures/operations using multiple assessors during a clinical placement, moves the validity and reliability of assessment methodology forward, beyond that offered by OpComp.

Simulators

The controlled environment of a skills centre permits reliable assessment of technical skills on simulations such as bench models, animals and computer models. However, their validity depends upon the fidelity of the simulation,75 with high-fidelity simulations possessing a high level of realism compared with a living human patient. Characteristics of high-fidelity simulations include visual and tactile cues, feedback capabilities and interaction with the trainee, with the opportunity for trainees to complete surgical procedures rather than isolated tasks. Although low-fidelity bench models and video box methods show less realism, these are often used to assess trainees because of lower cost, portability and the potential for repetitive use.

Parallel assessments of surgical skill, using live animals and bench simulations, have been compared by the Toronto group. 52 Performance was graded for both assessment formats using task-specific checklists, global ratings and pass/fail judgements (i.e. OSATS). Using 20 surgical residents, the correlations between live and bench scores were high (0.69–0.72), and the mean inter-rater reliability across stations ranged from 0.64 to 0.72. This study showed that using simulations with OSATS is a valid and reliable method of assessing surgical skill, with bench models giving equivalent results to live animal model simulations. Lentz et al. 76 repeated this comparison of bench and live animal assessments within a surgical laboratory curriculum and found that skills improved over time for individual trainees and as a cohort by year of training, suggesting that simulations can also be successfully used to assess progress during training.

Whenever possible, the assessment method used in the workplace should be used for the relevant simulation because this will aid validity and transferability. 77 Studies of the transferability of simulation-based assessments to actual operating theatre performance are of fundamental importance in establishing their validity as an assessment method. Several authors have demonstrated that assessments of technical skill on low-fidelity simulations predict performance in the operating theatre. 52,78–80 It does appear that complete procedures can be deconstructed into tasks suitable for trainees to rehearse and be assessed for competence, before moving on to surgical training in patients.

The more recent development of high-fidelity simulations for surgical assessment offers the potential to reduce assessor time while providing automatic ‘objective’ output data for feedback and assessment purposes. Output metrics for trainees performing virtual simulations include economy of motion, length of path movements and instrument errors. 81 Several studies have shown that both the minimally invasive surgical trainer–virtual reality (MIST-VR) (Mentice, Gothenburg, Sweden) and LapSim® Gyn VR systems (Surgical Science, Gothenburg, Sweden) are valid and reliable methods of assessing psychomotor skills for various laparoscopic procedures. 81–85 Furthermore, randomised controlled studies have demonstrated the benefits of high-fidelity simulation training for performing laparoscopic cholecystectomy,86,87 endoscopy88 and catheter-based interventions. 89

The main advantage of simulations is that they allow unlimited practice for trainees to learn a new procedure before moving to patients, thus providing a new opportunity to learn safely from mistakes. 90 Assessment using simulations provides access to surgical skill training and assessment in situations where training in patients is unavailable or very limited91 and/or too high risk. 92 Simulations can also be used to assess the performance of individuals within emergency teams93 and surgical teams. 94 It appears that simulators can take on a well-defined assessment role that complements the assessment of real-time operating theatre performance. However, practice and assessment on simulators are no substitute for real operating experience, although they offer trainees the opportunity to progress their surgical skills before training in the complex operating theatre environment.

Direct observation in the operating theatre

It seems axiomatic that direct observation of surgical performance in the operating theatre represents the ‘gold standard’ in terms of both content and construct validity. However, unstructured directly observed assessments suffer from halo error95 and other types of assessor bias. 73 In fact, the main issue with unstructured assessments using expert/consultant assessors is that the direct observation itself is not wholly successful, i.e. assessments are completed based on indirect observation or incomplete direct observation, which limits its reliability as an assessment method. 96

Standardising direct observation depends upon the use of a structured assessment, such as OSATS or PBA. Although the validity and reliability of structured direct observation has been comprehensively researched for simulations, the evidence for assessing surgical performance in the operating theatre is limited. This study seeks to directly fill this gap in the research.

There are two main types of rating scales used for structuring assessment of surgical skill: task analysis checklists and global ratings. These rating scales are compared in Table 2. Dual assessments using separate task analysis and global ratings may be time consuming to perform, but each method may have different roles. 53 Global ratings seem useful when assessing more complex operations, especially when there is more than one method of performing the task correctly, or when assessing experts for the purposes of certification or revalidation. Task analysis checklists provide a trainee with detailed instructions and feedback of how to undertake the operation in an approved way. The PBA and OSATS tools use both types of rating scales, in combination and separately respectively.

Procedure-based assessment

Procedure-based assessment is a method for assessing surgical skills in the operating theatre during interventional procedures. It is designed to be used in conjunction with the surgical logbook for all the index procedures for a particular surgical specialty. PBA was originally developed by the OCAP for trauma and orthopaedic surgery100 and PBAs have now been written for all other surgical specialties by the relevant specialty associations and SACs within the ISCP. Trainees already in training when PBA was introduced have been encouraged to use it, both as an aid to learning and to complement logbook experience, but the use of PBA has been made compulsory only for those entering surgical training since 2005 for orthopaedics and 2007 for the other surgical specialties. PBA has not yet been adopted by surgical training organisations outside the UK.

The assessment form itself has two principal parts. The first consists of a series of competencies within six core domains covering ‘consent’, ‘preoperative planning’, ‘preoperative preparation’, ‘exposure and closure’, ‘intraoperative technique’ and ‘postoperative management’. The consent and preoperative planning domains address perioperative competencies, whereas the remaining domains encompass intraoperative competencies. It is not expected that all PBA domains will be completed at any one time as consent and preoperative planning are often undertaken at a different time and place. While many of the competencies are common to all procedures (global items), others are specific to the particular procedure (task specific), particularly within the intraoperative technique domain. Each competency is assessed as satisfactory (S), unsatisfactory/development required (U/D) or not assessed (N). The assessment form is supported by a worksheet, originally used as part of the validation process, which gives examples of desirable and undesirable behaviours for each competency. Therefore, the first part of the PBA uses a combination of task and global items which are rated with a single binary rating scale. The second part of the assessment form consists of a four-level summary judgement in which the assessor rates the ability of the trainee to perform the observed elements of the procedure on that occasion with or without supervision (see Appendix 2). It uses similar levels to those used on the OpComp form. The content and construct validity of PBAs has been validated for index procedures in both general and orthopaedic surgery. 52,100

It is assumed that the assessor (clinical supervisor) will normally be scrubbed and supervising the trainee. Trainees carry out the procedure, or part of it, explaining what they intend to do throughout. The assessor will provide verbal prompts to remind the trainee to make explanations, if required, and will intervene if patient safety is at risk or the quality of treatment may be compromised. The form has been designed to allow the assessor to score items at the end of the procedure by using a simple binary rating scale. In addition, the completed form is intended to structure the provision of immediate constructive feedback (e.g. in the coffee room between cases). A PBA may be undertaken every time an index procedure is undertaken, as the primary aim is to aid learning.

The satisfactory standard for each competency is the level required for the CCT. At the end of a placement, a collection of PBAs, together with the logbook, will enable the educational supervisor or programme director to make a summary judgement about the competence of a trainee to perform an index procedure to the required standard.

Objective Structured Assessment of Technical Skills

The OSATS system was introduced by the RCOG (www.rcog.org.uk/education-and-exams/curriculum) as a formal requirement of their new training and education programme for all grades of trainees since August 2007. Prior to this, OSATS was used informally within training, particularly for junior O&G trainees.

The development of OSATS by Reznick et al. 52,101 at the University of Toronto in the 1990s initiated the current trend towards structured observational assessment. OSATS was originally developed for use on bench model simulations and was designed to be completed in real time by assessors, as with objective structured clinical examinations, rather than at the end of the procedure. OSATS has been shown to possess good inter-rater reliability and construct validity for assessing general surgical trainees performing common operations on both cadaver and live animal simulations. 52 Goff et al. 102 have also demonstrated that OSATS possesses construct validity and good inter-rater reliability for blinded and unblinded assessment of O&G trainees performing common procedures on lifelike models in a multiple station exam format. Direct observation or videoing of real surgical procedures in the operating theatre using structured checklists based on OSATS can demonstrate high inter-rater reliability and construct validity for simple operations such as varicose veins surgery. 53,103

The original OSATS was developed by Winckel et al. 101 and consisted of a technical checklist (rated on a numerical scale using 0 for ‘not performed’, 1 for ‘performed poorly’ and 2 for ‘performed well’) that was specific for the procedure. The second part was a generic assessment of 10 global items (using a five-point numerical rating scale from 0 ‘poorly, or never’ to 4 ‘excellent or always’) that were common to all procedures. The global rating assessment was modified by Martin et al. ,52 reducing the number of global items to seven, changing the five-point numerical rating scale to a behaviourally anchored scale using descriptors (e.g. ‘makes many unnecessary moves’ to ‘fluid moves with instruments and no awkwardness’ for global item time and motion) and including a pass/fail judgement at the end of the global assessment. Therefore, the OSATS form uses separate task and global assessments which are rated using two different rating scales, with a summary pass/fail judgement.

The OSATS adopted by the RCOG for the assessment of 10 O&G index procedures uses a similar form to the modified Martin et al.’s OSATS version. 52 Part 1 is a technical checklist in which task items, specific to the index procedure, are rated as ‘done independently’ or ‘needs help’. Part 2 provides the generic assessment, which has been reduced to a three-point behaviourally anchored rating scale of seven global items (see Appendix 1). These seven global items have been further modified by the RCOG to combine some global items (e.g. ‘instrument handling’ and ‘knowledge of instruments’ have been combined as ‘knowledge and handling of instruments’), whereas other global items are new additions (e.g. ‘suturing and knotting skills’, ‘relations with patient and the surgical team’, ‘insight/attitude’ and ‘documentation of procedures’). This is combined with a summary pass/fail judgement (this version was in use at the time of this study). However, the pass/fail terminology has since been revised to ‘competent in all areas included in this OSATS’ and ‘working towards competence’ to enforce the formative nature of the OSATS. 104 For OSATS to count as a pass, an assessment algorithm is used: all items on the technical checklist must be ticked as ‘performed independently’, and the majority of global items ringed in the middle to right of the rating scale, while insight/attitude must be consistently ringed ‘fully understands areas of weakness’. 105 Trainees are required to achieve a set number of passes for each index procedure in order to ‘sign off’ their logbook competency for that procedure. Currently, the requirement is for three completed OSATSs by at least two trainers in order for the relevant logbook competency to be signed off as competent for independent practice. Validity and reliability studies have not been undertaken for the application of this revised tool for assessing trainees in the operating theatre environment.

From our literature review and from email correspondence with surgical training organisations in North America, Continental Europe and Australasia, it appears that OSATS is not routinely used within any surgical curriculum in the workplace.

Non-technical Skills for Surgeons

The NOTSS system is a behavioural rating system developed using a multidisciplinary group of surgeons, psychologists and an anaesthetist from the Royal College of Surgeons of Edinburgh, in collaboration with the University of Aberdeen. 106 The development of NOTSS built on work from a similar project that developed a behaviour rating system for anaesthetists called Anaesthetist’s Non-technical Skills (ANTS). 107 While ANTS is not used formally in UK anaesthetic training, it has been piloted for use within the Australian and New Zealand College of Anaesthetists training programme. NOTSS is not currently part of the curricula for surgeons in training in the UK. However, there are a number of ongoing process trials worldwide that are considering the adoption of NOTSS. Furthermore, the Royal Australian College of Surgeons has adapted and expanded NOTSS to establish a surgical performance framework for assessment purposes. 108

Behavioural rating systems are already used to structure training and evaluation of non-technical skills in anaesthesia, civil aviation and nuclear power, to improve safety and efficiency. They are rating scales based on skills taxonomies, with examples of good and poor behavioural markers, and are used to identify observable, non-technical behaviours that contribute to superior or substandard performance. Behavioural rating systems are context specific and should be developed in the domain in which they are to be used.

Non-technical Skills for Surgeons describes the main observable non-technical skills associated with good surgical practice. It has been designed to provide surgeons with explicit ratings and feedback on their non-technical skills, either within the operating theatre or operating theatre simulator. The system comprises only behaviours that are directly observable or can be inferred through communication during the intraoperative (‘gloves on, scrubbed up’) phase of surgery.

The NOTSS system comprises a three-level hierarchy consisting of categories (at the highest level), elements, and behaviours. Four skill categories and 12 elements make up the skills taxonomy (see Appendix 3). Each category and element is defined with examples of good and poor behaviours for each element. These exemplar behaviours were generated by consultant surgeons and are intended to be indicative rather than comprehensive. The aim is to provide a common terminology and assessment framework for surgical trainees and consultants to structure their training needs, in order to develop their non-technical skills in the workplace. 106

The development and design of the NOTSS system has been a rigorous and structured process, from initial task analysis through to system evaluation. For more details on NOTSS development see Yule et al. 109 The psychometric evaluation of NOTSS to date has been carried out by the NOTSS development team in the operating theatre simulator. 110 Six video scenarios were designed and filmed by practising surgeons, anaesthetists and nurses with experience in non-technical skills, to illustrate a range of surgeons’ non-technical skills. A group of 44 consultant surgeons, trained in the use of NOTSS, observed and rated these videos using NOTSS assessment forms. Their ratings were compared with expert ratings for ‘accuracy’ and assessed for inter-rater reliability. In this study, the NOTSS system demonstrated a consistent internal structure, and internal reliability was high for all four categories (overall mean difference of 0.25 scale points between categories and elements). The sensitivity (‘accuracy’) of the system was moderate (mean sensitivity across all categories was 0.67 scale points difference from expert ratings) with the ‘decision-making’ category most sensitive and the ‘situation awareness’ category least sensitive. The inter-rater reliability (mean within-group agreement of ratings across NOTSS categories) was acceptable (> 0.7) for the categories ‘communication and ‘teamwork’ and ‘leadership’, although below acceptable for categories ‘situation awareness’ and ‘decision-making’.

The NOTSS development team has analysed the level of agreement between ratings of expert versus novice raters in more detail. 111 The mode ratings of NOTSS category ratings for each video scenario showed 50% agreement. Where there was disagreement, novice raters were more likely to give harsher ratings, with the widest differences between rater groups within ‘communication and teamwork’ and ‘leadership’ domains. Of note, 23% of ‘situation awareness’ ratings were scored not applicable by novice raters, compared with 0% of experts, indicating that experts in non-technical skills are more equipped to rate these behaviours. This highlights the role of training in human factors training and the rehearsal of non-technical skill ratings for NOTSS assessors.

The NOTSS development team has also considered the usability of the NOTSS system112 within the operating theatre for observing common surgical procedures. Questionnaire responses from consultant surgeons participating in the usability trial have indicated that NOTSS provides a structure and language to rate and provide trainee surgeons with feedback on their non-technical behaviours. The main concerns with using the NOTSS system were reported as: difficulty in understanding the behavioural descriptors; difficulty in rating behaviours when trainees do not verbalise adequately; assessments being more suitable for senior than for junior surgical trainees; the use of routine cases raising insufficient decisions for rating behaviours in the decision-making category; and the ‘negative’ impact of a scrubbed consultant upon trainee-led leadership. These user-satisfaction responses from NOTTS assessors provide great insight into the challenges of rating behaviours in the operating theatre.

This study moves forward the psychometric evaluation of the NOTSS system into the operating theatre environment. The tool will be used for observing and rating surgeons’ non-technical skills in vivo. In addition, this study provides further work on the usability of the NOTSS system in the operating theatre.

Specialties and index procedures

The index procedures were selected by the research team in collaboration with each surgical specialty and after subsequent approval by the Steering Committee (Table 3).

The index procedures represented typical procedures for each specialty that were performed on a regular basis, allowing for frequent assessments over a spread of trainee grades. They were also chosen to reflect a range of procedural difficulty/complexity and the breadth of surgery in each specialty, e.g. open and laparoscopic procedures.

Assessment tools and questionnaires

Sampling

The sampling aim was that at least two assessors would assess each surgical trainee undertaking each index procedure in their specialty on at least two occasions, equating to a minimum of eight assessments per trainee in those specialties with two index procedures and more assessments per trainee in those specialties with additional index procedures. This sampling strategy was designed to allow the estimation of variation in trainee performance between individual cases and types of index procedure and differences in case complexity, as well as variability in assessor stringency and subjectivity. Furthermore, the procedures would be assessed as close together as possible for an individual trainee to avoid any significant training effect. In this way, the study methodology was orientated to provide performance-focused assessments most suited to reliability analysis.

This sampling grid guided the sampling plan but, during implementation, a pragmatic approach to recruitment had to be taken, depending on which surgical trainees and clinical supervisors were available for assessments of suitable index procedures.

Videos of cases were recorded where patient consent was given to do so and when it was practically possible to film the case. Two cameras were used to record an operating theatre view and an operative field view. During laparoscopic procedures, the operative field view was swapped to the laparoscope camera view. The two images were recorded screen-in-screen on to a hard-disk recorder together with dual sound recordings of the clinical supervisor and trainee, who wore radio microphones.

Sample size

Generalisability theory provides a reliability estimate. It is not a hypothesis test and does not therefore include an accepted approach for power calculation. However, to produce reliable estimates it is essential to sample each relevant factor (trainees, case, assessors, etc.) as widely and representatively as possible.

The proposal at the start of the study was to recruit 50–60 surgical cases for each index procedure, which gave a total of 450–540 cases. In view of initial difficulties with recruitment and the addition of further index procedures, the Steering Committee recommended a total of at least 300 cases with a good spread across specialties and index procedures. Although this number was seen as a minimum requirement for the overall study, it was increased in light of the need to compare PBA and OSATS within O&G. An overall target of 450 cases was subsequently set, of which 150 were intended to be within O&G.

Recruitment, study context, and user satisfaction and acceptability

Numerical data in the form of total numbers, medians and averages are presented using frequency tables. They were calculated directly from responses on structured questionnaires and are presented as proportions of those who responded. No statistical tests were applied as the sample sizes were too small. Therefore, comparative statements are used purely to explain observed differences in our questionnaire response data. Box-and-whisker plots are used to illustrate any differences between clinical supervisor and trainee perspectives. Qualitative data from the same questionnaires have been collected and stored but not yet analysed.

Reliability

Reliability is a measure of precision and discrimination. Broadly, it reflects the reproducibility of assessment rankings. In the context of this study, reliability represents how well a trainee’s score based on x assessors observing y cases would predict that same trainee’s score if a different x assessors observed a different y cases. More exactly, it represents how two different trainees would score relative to one another if they were both dual assessed according to that pattern.

If we call the stable differences between trainees ‘true variance’, then the impact of case variation, assessor variation and other sources of unwanted variation might be called ‘error variance’. Reliability (R) is given by the equation:

The coefficient R will be a fraction between 0 and 1, where 0 is the worst possible reliability (all error) and 1 is the best possible reliability (all true). A reliability coefficient of ≥ 0.7 is generally accepted as sufficient for ‘low-stakes’ assessment situations; ≥ 0.8 is generally accepted as sufficient for ‘high-stakes’ assessment situations.

We calculated reliability using generalisability theory. 121 Generalisability studies apply variance component analysis to estimate the impact on an assessment score of every relevant factor (G-study). They then combine the sources of variance using equations derived by Cronbach to model the reliability coefficient in a given assessment situation with a particular number of assessors and cases (D-study). A modelled reliability coefficient is thus called a generalisability coefficient (G).

Our G-study used the VARCOMP procedure. We selected the MINQUE method because of its superior handling of unbalanced data. 122 Our regression model assumes that each factor contains a random sample from an infinite universe and estimates the factors in Table 4.

Our D-studies are reported in tabular form to show how G varies with increasing numbers of cases and assessors. To mimic real assessment formats the axes of the table are ‘cases’ and ‘assessors per case’. This means that (for example) cell 2,2 represents the reliability of the scores from four different assessors, two observing one case and two observing another. The tables assume that judges are drawn from a similar mix of designations in every trainee’s assessment.

We have modelled reliability for two situations. Firstly, we have estimated the reliability of a format where trainees all perform the same index procedure. The D-studies for comparing trainees within a procedure use the formula:

where N means ‘the number of’.

Secondly, we have estimated the reliability of a format whereby trainees all perform an identical mix of two procedures. The D-studies for these tables use the formula:

Validity

Validity indicates how well the score reflects the intended construct of ‘surgical performance’. The study provides many sources of information about validity and these will all be presented in evidence for or against the validity of the three assessment methods. If valid, the following hypotheses will be fulfilled:

1. Scores obtained by each assessment will correlate with the other assessments that set out to measure the same aspect of performance. These correlations will operate within instruments (internal structure) and between instruments.

2. Scores will increase with duration of surgical training and number of similar procedures performed (experience).

3. Higher-scoring operations will result in less operative time and blood loss, fewer perioperative and postoperative complications and a shorter length of hospital stay.

4. Mean scores, and scores for each element, will not be significantly different across the fifteen different index procedures.

5. Assessor designation (clinical supervisor versus independent assessor for PBA and OSATS, anaesthetists and scrub nurses versus independent assessor for NOTSS) will not affect assessment stringency.

6. Assessment (plus or minus video-recording equipment) will not affect the performance of trainees.

Each of these hypotheses will be tested as follows: Pearson’s method will be used for hypotheses 1, 2 and 3. In addition, for hypothesis 1, factor analysis will test internal structure (principal axis factoring with varimax rotation). For hypothesis 2, categorical regression (CATREG procedure) will regress to correct for intercorrelations between predictor variables. Regarding hypothesis 4, the raw mean scores for each procedure are confounded because they are based on very different samples of trainees and assessors in each case. Thus the best test of whether the different procedures themselves produce different scores is to examine the value of Var(proc) in the G-study tables for each instrument. Var(proc) represents the ‘consistent differences between index procedures after correcting for the different samples of trainees doing, and assessors assessing, each index procedure’.

Exactly the same considerations apply to hypothesis 5. The raw mean scores for each assessor designation are confounded by the uneven sampling of individual assessors and of trainees and procedures. The value of Var(designation) in the G-study tables, however, corrects for this and represents ‘that part of Var(assessor) that can be explained by designation on the basis that it is consistent within a group of assessors of the same designation’.

For hypothesis 6, we do not have the data to compare the performance of ‘not assessed’ and ‘assessed’ cases. We therefore present the questionnaire-based perceptions of trainees and supervising surgeons as a proxy indicator. In addition, we compare the scores given to those cases in which either party felt that assessment affected performance with the scores given with those cases in which neither party perceived an effect. We also compare the scores given to video-recorded and non-video-recorded cases. Both comparisons use unpaired t tests.

Where appropriate, the Bonferroni correction was used to raise the threshold for significance where several outcomes were used to evaluate the same hypothesis.

An interim analysis was performed at 1 year and submitted to the Steering Committee. After discussion with the Steering Committee, a post hoc analysis was performed to explore the reasons for differences in the reliability of the PBA and OSATS tools.

Patients

All patients had to be provided with written information in advance of their informed consent prior to their recruitment within the study. Information was sent to 832 patients whose surgery was identified as a suitable training case. A total of 274 (33%) of these patients were not consented for the study. A small proportion declined consent (n = 14), but the vast majority (n = 260) were not approached for consent because of lack of availability of an inpatient bed, alteration or cancellation of the operating list, or known non-availability of any trainee or research team member for the operating list. Of the 558 patients who were consented for participation in the study, 121 (22%) were not included. The flowchart in Figure 5 illustrates the percentages of consented and included patient cases.

FIGURE 5.

Flowchart of cases consented and included.

Table 5 displays the number of cases consented, included and not included for each specialty. The greatest numbers of patients included were from O&G (n = 183) and vascular (n = 91) specialties, while the lowest numbers were from colorectal (n = 25) and orthopaedic specialties (n = 36).

The percentages of included versus non-included cases (i.e. lost to assessment) for each specialty are shown in Figure 6. Orthopaedic (31%) and cardiac (27%) surgery had the largest proportion of lost cases.

FIGURE 6.

Percentage of included versus non-included cases for each surgical specialty.

We recorded the reason for non-inclusion in each case. Fourteen cases were not included because no member of the research team was available to co-ordinate the assessments in theatre. Within O&G surgery, 29 cases were lost because a provisionally planned caesarean section did not proceed. These cases were excluded from further analysis as we wanted to accurately reflect the clinical obstacles to WBA for trainees. The most common reasons for non-inclusion across all surgical specialties were no trainee available to perform the case (27%), the clinical supervisor performing the case, either because he or she felt the case was unsuitable for training the trainee available or because he or she chose to perform the case personally (22%), and no list time available for surgical training (12%). The contribution of each factor to the loss of cases is illustrated in Figure 7.

FIGURE 7.

Reasons for lost assessment cases.

Assessor/trainee consent and participation

Having received information about the study and an invitation to participate, all clinical supervisors and all but two trainees within the six surgical specialties, while the study was active in those specialties, gave verbal consent to participate in the study. The two trainees who declined to consent were within O&G and were near achieving their CCT. Verbal consent from these groups was sought before each round of recruitment in that specialty or when an individual entered the specialty if this occurred part-way through the recruitment round. Four clinical supervisors (two in orthopaedics, one in cardiac surgery and one in colorectal surgery) did not go on to undertake any assessments because of logistical reasons which have been identified above. Six trainees (two in orthopaedics and one in colorectal, one in cardiac and two in upper GI surgery) did not go on to be assessed, also for logistical reasons. All anaesthetists, nurses and surgical-care practitioners (SCPs) working within the operating theatres were given prior information about the study and invited to participate. Individual assessors were approached for verbal consent as far in advance of their first case as possible. However, these professional groups sometimes rotated through the operating theatres in an unpredictable manner and it was not always possible to seek consent until the morning of the list. Owing to the large numbers of personnel within these groups, verbal consent was sought only from those individuals attached to the relevant operating lists and not from all potential recruits as was the case with clinical supervisors and trainees. No nurses and only one anaesthetist who were approached to participate declined to do so. Numbers of non-recruited anaesthetists and nurses were not recorded. Demographic information was not collected for any non-participant assessors or trainees as they had not given consent for us to do so.

Surgical cases

Fifty-one clinical supervisors, 56 anaesthetists, 39 scrub nurses, two SCPs and four independent assessors provided 1635 assessments on 85 trainees undertaking 437 cases. The distribution of assessments for these surgical cases using the three assessment tools is shown in Table 6. A total of 749 PBAs, 695 NOTSS assessments and 191 OSATS were performed. There are many more assessments than cases owing to the multiple assessments provided by different assessor groups. In addition, the independent assessor completed an OSATS/PBA and NOTSS assessment on many occasions. Independent assessor (IA)1 completed more than one assessment tool in 75% of cases, IA2 in 73% of cases and IA3 in 86% of cases.

Assessor and trainee demographics

Surgical trainee demographics

Of 85 surgical trainees, 82 provided almost complete demographic data (98% response rate) and there are some data for all trainees. The descriptive statistics are presented as a proportion of those who responded (this is also the case for presentation of the assessor demographic data). The greatest share of trainees were from O&G (38%), while the remaining trainees were fairly evenly split between the other five surgical specialties (Figure 8).

FIGURE 8.

Proportion of trainees in each surgical specialty.

The trainees’ demographics are summarised in Table 7 and illustrated in Figures 9–13. The majority of all trainees were male (65%). Male gender predominated in all specialties (67%–100%), with the exception of O&G, where 78% were female; 100% of all trainees in cardiac and orthopaedic specialties were male.

TABLE 7 - Trainee demographics by surgical specialty

		Cardiac	Colorectal	Upper GI	Orthopaedic	O&G	Vascular
Total number	Frequency	10	9	13	9	33	11
	%	12	10.5	15	10.5	39	13
Number males	Frequency	10	6	12	9	9	9
	%	100	67	92	100	28	90
Age (years)	Median	37	33	33	35	32	34
	Range	28–45	28–39	27–43	34–40	25–40	27–46
UK graduate	Frequency	1	7	9	7	13	6
	%	10	78	69	78	41	60
ST level	Median	8	3	3	8	3	3
	Range	3–8	3–8	1–7	3–8	0–7	2–7
Total years’ surgical training	Median	11	6	9	9	5.5	6
	Range	4–17	2–13	0–14	4–18	0–15	2–19
Years’ UK surgical training	Median	11	6	4	9	3	5
	Range	1–13	2–12	0.2–12	4–12	0.5–9	2–10
Years’ non-UK surgical training	Median	3	2.5	0	0	0	0
	Range	0–6	1–4	0–7	0–7	0–6	0–9
Total index procedures	Median	57.5	7	30	116	46	3.5
	Range	0–450	0–54	0–200	5–350	0–400	0–500
Recent index procedures	Median	13.5	4	17.5	15	10	1.5
	Range	0–60	1–10	0–58	3–60	0–45	0–20

FIGURE 9.

Percentage of male and female trainees by surgical specialty.

FIGURE 10.

Percentage of UK and non-UK graduates by surgical specialty.

FIGURE 11.

Median age (years) and ST level of trainees by surgical specialty.

FIGURE 12.

Medians for total years of surgical training, UK surgical training and non-UK surgical training by surgical specialty.

FIGURE 13.

Medians for total number and number of recently performed index procedures prior to taking part in the study by surgical specialty.

Half of all trainees graduated in the UK (51%), although the proportion of UK and non-UK graduates in each specialty differed substantially. Colorectal and orthopaedic specialties had a large majority of UK graduates (both 78%), whereas cardiac trainees were predominantly non-UK graduates (89%) with the highest median for number of years of non-UK surgical training.

The most experienced trainees, in terms of age, total years of surgical training and total number of index procedures previously performed, were from the cardiac and orthopaedic specialties. Trainees in vascular surgery had the greatest range in years of total and non-UK surgical training (2–19 years and 0–9 years respectively) and the greatest range of previous total number of index procedures performed (0–500).

All levels of ST were represented within the trainees recruited across the study. The highest median ST level was within cardiac and orthopaedic specialties.

The spread of trainees according to their year of ST within each of the surgical specialties is presented in Figure 14. The percentage of trainees at each ST level is presented in Figure 15. The greatest percentage of trainees was at either a junior level (ST2 or ST3) or a senior level (ST7 and ST8), reflecting the tertiary referral centre/teaching hospital setting for the study.

FIGURE 14.

Trainees by specialty and year of training.

FIGURE 15.

Percentage of trainees at each ST level.

Clinical supervisor demographics

Of 51 clinical supervisors, 50 provided almost complete demographic data (98% response rate) and there are some data for all clinical supervisors. The greatest share of clinical supervisors were from O&G (37%), while the remainder were fairly evenly split between the other five surgical specialties (Figure 16).

FIGURE 16.

Percentage of clinical supervisors in each surgical specialty.

The majority of clinical supervisors were male (90%) and were UK graduates (78%). There was a broad age range across the specialties (37–61 years), although the median age was similar within each specialty (45–48 years). There was also a broad range of experience (1–26 years) across all specialties. The median years of experience was highest in O&G (13 years) and cardiac (10.5 years) and lowest in upper GI (5 years). Six senior trainees acted as assessors in a small number of cases (n = 15, 3% of total). Their demographic data are not included in Table 8 but rather in the demographic data for trainees.

TABLE 8 - Clinical supervisor demographics by surgical specialty

		Cardiac	Colorectal	Upper GI	Orthopaedic	O&G	Vascular
Total number of clinical supervisors	Frequency	6	5	7	5	19	9
	%	12	9.5	14	9.5	37	18
Age (years)	Median	47	48	45	45	47	45
	Range	43–53	43–61	38–60	39–51	37–60	40–52
Number males	Frequency	6	3	7	5	13	9
	%	100	75	100	100	76	100
UK graduate	Frequency	5	3	6	5	12	7
	%	83	75	100	100	63	78
Experience (years as a consultant)	Median	10.5	6.5	5	9	13	8
	Range	8–18	2–8	1–26	3–16	1–25	1–17

Procedure-based assessment – prior experience

Procedure-based assessment questionnaires were given to all clinical supervisors and trainees in the six surgical specialties. Questions relating specifically to previous experience with the PBA tool were excluded from questionnaires for O&G clinical supervisors and trainees. PBA was introduced to O&G for the purposes of our study. Therefore, they would not have used the tool before.

Twenty-eight of the 38 non-O&G clinical supervisors (response rate 74%) provided almost complete responses about their prior experience and training in using the PBA tool for assessment and feedback. They reported a range of prior experience and a range of training (Table 10). Overall, 39% of them reported that they had not assessed trainees using PBA prior to their involvement in the study. However, where PBA had been used in WBA, 44% them had given feedback to trainees as part of the assessment process.

Forty-one of 53 non-O&G trainees (response rate 77%) provided responses about their experience of being assessed and receiving feedback using PBA. They reported a range of prior experience (Table 11). Forty-nine per cent had never previously been assessed with PBA.

Procedure-based assessment – training

Forty-four of 52 clinical supervisors (response rate 85%) provided almost complete responses about their PBA training. The questionnaires asked them about the types of PBA training they had received and the overall adequacy of this training for using PBA (Table 12). Eighty-nine per cent of them reported that they had undergone some form of training and 70% agreed that the training had been adequate. Clinical supervisors were asked about the different types of training received, and multiple responses were possible. Face-to-face and written information (60% of clinical supervisors) were the most common types of training, reflecting the training approach adopted by the research team, with 20% of clinical supervisors having received a combination of two or more types of training.

Sixty-six out of 85 trainees (response rate 78%) provided almost complete responses about their PBA training (Table 13). Sixty-five per cent of them reported that they had received some form of training and 41% agreed that they had had adequate training. A greater proportion of trainees accessed the ISCP web guidance for training, although face-to-face and written information remained the most common types of training.

Objective Structured Assessment of Technical Skills – prior experience

Objective Structured Assessment of Technical Skills questionnaires were given to all clinical supervisors and trainees in O&G. As OSATS had been used informally in clinical practice for 2 years before the study began, prior experience with OSATS was assessed according to length of time used.

Seventeen of 21 O&G clinical supervisors (response rate 81%) provided responses about OSATS (Table 14). There was high previous experience of using the OSATS tool. Only 6% of them had never previously assessed a trainee using OSATS prior to their involvement in the study.

Twenty-eight of 33 O&G trainees using OSATS (response rate 85%) provided responses about the method (Table 15). They reported similarly substantial experience in previous use of OSATS to their O&G clinical supervisors, with only 4% never having been assessed with OSATS before.

Objective Structured Assessment of Technical Skills – training

Sixty-four per cent of O&G clinical supervisors who responded were satisfied with their training in the use of the OSATS tool (Table 16). This is similar to the proportion of the total clinical supervisor cohort who were satisfied with their training in the use of PBA (70%).

The O&G trainees were less satisfied than their O&G clinical supervisors with training for the use of OSATS (43% vs 64%) (Table 17). This is also similar to the proportion of the total trainee cohort who were satisfied with their training in the use of PBA (41%). Both groups had accessed similar training methods, and the same proportion of both groups (29%) had used a combination of training methods.

Non-technical Skills for Surgeons

Non-technical Skills for Surgeons questionnaires were given to all anaesthetist and scrub nurse assessors. NOTSS assessments are not in current training use and therefore questions regarding prior experience with this tool were not relevant. The training methods outlined reflect the training received for the study’s purpose alone.

Thirty of 56 anaesthetists (response rate 54%) provided responses about the NOTSS tool. The anaesthetists reported a range of perceived training (Table 18). Responses on whether the adequacy of training was sufficient were evenly divided between agreement and disagreement.

Twenty-six of 39 nurses (response rate 67%) provided almost complete responses about using NOTSS. They reported a range of perceived training (Table 19). Eighty-four per cent of them reported having received face-to-face training compared with 63% of anaesthetists. The majority of them (82%) felt that the NOTSS training received was adequate compared with 47% of anaesthetists.

Procedure-based assessment user satisfaction and acceptability

Forty-four of 52 clinical supervisors (response rate 85%) provided almost complete responses about their satisfaction with the PBA assessment and feedback process. They provided mixed, but predominantly positive, responses about the use of the PBA tool (Table 20). The majority of them agreed or strongly agreed that PBA was valuable as a tool for providing feedback (77%), for formative assessment (72%), for summative assessment (68.5%), and to support reflective practice (63%). Of those clinical supervisors who strongly disagreed or disagreed that PBA was a valuable tool for assessment, more disagreed that it was useful in summative assessment than formative assessment (15.5% vs 6%).

Clinical supervisors were asked to report their opinions of PBA based upon personal use of the tool. A categorical scale from 0 to 10 was used for these questions where 0 was ‘not at all’, 5 was ‘moderately’ and 10 was ‘very much’ (Table 21 – median scores are shaded). The clinical supervisors’ overall satisfaction with PBA was moderately good. They reported that they thought trainees found their feedback moderately useful (median score 6, interquartile range 5–7). The extent to which PBA enhanced their ability to assess was reported as moderate (median score 5, interquartile range 4–7), although 7% reported that it did not help them at all to assess trainees (score 0). Most clinical supervisors felt comfortable reporting their concerns about a trainee’s surgical skills on a PBA form (median score 7, interquartile range 4.5–8). Two questions were asked only of non-O&G clinical supervisors, for whom PBA is the current WBA tool within the ISCP. These questions related to the role of PBA in surgical education and whether the clinical supervisor would use PBA in the future if given the choice. The majority of non-O&G clinical supervisors felt that PBAs were moderately important in surgical education (median score 6, interquartile range 5–7). However, there was a wide range of opinion as to whether, if given the choice, they would use PBA in the future (median score 6, interquartile range 3.5–8). The majority (68%) felt that they were more likely than not to choose to use PBA in the future (scores ≥ 5).

Sixty-six of 85 trainees (response rate 78%) provided almost complete responses about their satisfaction with PBA and its feedback process. They provided mixed, but predominantly positive, responses about the use of the PBA tool (Table 22). The majority of trainees agreed that PBA was useful for providing feedback (88%), for formative assessment (72%), for summative assessment (64%), and to support reflective practice (74%). In contrast to clinical supervisor responses, there was slightly more agreement/strong agreement among trainees for the use of PBA as a formative assessment than for its use as a summative assessment (72% vs 64%). In keeping with clinical supervisors’ responses, more trainees disagreed or strongly disagreed with the use of PBA for summative assessment than for formative assessment (16% vs 8%). A greater proportion of trainees than clinical supervisors felt that PBA was useful for providing feedback (88% vs 77%) and to support reflective practice (74% vs 63%).

Trainees were also asked to report their opinions of PBA, based upon personal use of the tool (Table 23 – median scores are shaded). Trainees reported that they found the feedback provided by the clinical supervisors moderately to very useful (median score 7, interquartile range 6–8). The extent to which trainees felt that PBA had enhanced the ability of their clinical supervisors to assess them was reported as moderate (median score 6, interquartile range 4.5–7), although 6% of trainees felt that it did not help their clinical supervisors at all to assess them (score 0). Most trainees, like the clinical supervisors, felt comfortable with clinical supervisors reporting their concerns about a trainee’s surgical skills on a PBA form (median score 7, interquartile range 5–8). Non-O&G trainees were asked the same two additional questions as non-O&G clinical supervisors. The majority of non-O&G trainees felt that PBA was moderately to very important in surgical education (median score 7, interquartile range 5–8). The vast majority of trainees (84% with scores ≥ 5) reported that, if given the choice, they would be likely to choose to use PBA in the future (median score 8, interquartile range 6–9).

Key differences in the responses of clinical supervisors and trainees regarding their experiences in using PBA are illustrated by box-plots (Figures 17–20), with median scores displayed by a vertical black line, the interquartile range of scores by the grey box and the range of scores by the horizontal whisker lines, with outliers identified as small numbered circles. There were differences of opinion regarding the value of the PBA feedback process, with trainees finding the feedback provided more useful than the clinical supervisors themselves felt it was useful to trainees. Trainees also perceived that PBA enhanced the process of assessing surgical skills more than clinical supervisors did. Trainees placed more value upon the importance of PBA within surgical education than did clinical supervisors. The most striking difference of opinion was regarding the likelihood of choosing to continue using PBA in the future as a method of assessing surgical skills.

FIGURE 17.

Comparing clinical supervisors’ and trainees’ opinion of the usefulness of feedback with PBA.

FIGURE 18.

Comparing clinical supervisors’ and trainees’ opinion of enhancement of the assessment process with PBA.

FIGURE 19.

Comparing clinical supervisors’ and trainees’ opinion of the importance of PBA in surgical education.

FIGURE 20.

Comparing clinical supervisors’ and trainees’ opinion of the likelihood of using PBA in the future if given the choice.

Objective Structured Assessment of Technical Skills user satisfaction and acceptability

Seventeen of 20 O&G clinical supervisors using OSATS (response rate 85%) provided almost complete responses about their satisfaction with OSATS and its feedback process. They provided predominantly positive responses about the use of the OSATS tool, but the greatest number of negative responses were over its summative use (Table 24). The majority of them felt that OSATS was valuable as a tool for providing feedback (88%), for formative assessment (76%), for summative assessment (59%), and to support reflective practice (71%). There was a clear difference in agreement or strong agreement for the use of the OSATS tool as either a formative or a summative assessment (76% vs 59%), with more trainers disagreeing or strongly disagreeing with the use of OSATS as a summative assessment than as a formative assessment (24% vs 6%). In addition, the majority who responded felt that OSATS did not add time to their operating list (66%).

Overall, O&G clinical supervisors’ satisfaction with OSATS was moderately good (Table 25). Clinical supervisors reported the extent to which OSATS enhanced their ability to assess as moderately good (median score 7, interquartile range 5–7). Their overall satisfaction with OSATS as an assessment tool was moderately good (median score 7, interquartile range 4.5–7.5). However, the 15 O&G clinical supervisors who had used both OSATS and PBA expressed a slightly greater overall satisfaction with PBA as an assessment tool (median score 7, interquartile range 5–9). In response to the direct question ‘Do you prefer OSATS or PBA?’, five (29%) preferred PBA, two (12%) preferred OSATS and 10 (59%) expressed no preference.

Twenty-eight of 33 O&G trainees using OSATS (response rate 85%) provided almost complete responses about their satisfaction with OSATS and its feedback process. They provided mixed responses about the use of OSATS (Table 26), which were less positive than those of their clinical supervisors. Similarly to the clinical supervisors’ responses, the greatest number of negative responses related to its summative use. The only response that was answered positively by the majority of trainees (83%) was regarding the value of OSATS in providing feedback after an operation. Fifty per cent were of the opinion that OSATS is valuable as a formative assessment tool and 36% as a summative assessment tool. Almost half (46%) felt OSATS was a useful tool to support reflective practice.

The overall satisfaction of O&G trainees with OSATS was not particularly high (Table 27). The extent to which trainees felt that OSATS had enhanced the ability of their clinical supervisor to assess them was reported as moderate (median score 6, interquartile range 5–7). Their overall satisfaction with OSATS as an assessment tool was just moderate (median score 5, interquartile range 3.25–6.75). Directly comparing the overall satisfaction of trainees with clinical supervisors, 43% versus 65%, respectively, were satisfied with OSATS as an assessment tool (scores ≥ 6). However, the 24 trainees who had used both OSATS and PBA expressed a greater overall satisfaction with PBA as an assessment tool (median score 6.5, interquartile range 4–7.75). In response to the direct question ‘Do you prefer OSATS or PBA?’, 11 (39%) preferred PBA, five (18%) preferred OSATS and 12 (43%) expressed no preference.

Key differences in the responses of clinical supervisors and trainees regarding their experiences in using OSATS are illustrated in box-plots (Figures 21–23). Clinical supervisors perceived that OSATS enhanced the process of assessing surgical skills more than trainees. In addition, clinical supervisors expressed greater overall satisfaction than trainees with both OSATS and PBA as assessment tools. The O&G pattern of responses for OSATS, with O&G clinical supervisors’ user satisfaction greater than that of O&G trainees, contrasts with the PBA user satisfaction results, in which non-O&G trainees reported greater satisfaction with PBA than their clinical supervisors.

FIGURE 21.

Comparing O&G clinical supervisors’ and trainees’ opinion of enhancement of the assessment process with OSATS.

FIGURE 22.

Comparing O&G clinical supervisors’ and trainees’ opinion of overall satisfaction with OSATS as an assessment tool.

FIGURE 23.

Comparing O&G clinical supervisors’ and trainees’ opinion of overall satisfaction with PBA as an assessment tool.

Non-technical Skills for Surgeons user satisfaction and acceptability

Thirty of 56 anaesthetists (response rate 54%) provided almost complete responses about the NOTSS tool (Table 28). More agreed that they were able to easily assess interpersonal skills than cognitive skills using NOTSS (60% vs 27%). A majority agreed that the tool was useful for reflection (73%) and that it added value to the use of surgical skill assessments (60%). All of the 46% of anaesthetists who responded to this question felt NOTSS was useful for providing feedback. However, they were evenly split on whether or not it would enhance patient safety. Only one anaesthetist felt that using NOTSS added too much time to the list, but anaesthetists declined to complete a NOTSS form on some occasions because they said they were too occupied with their clinical duties at the beginning of the procedure.

Twenty-six of 39 nurses (response rate 67%) provided complete responses about using NOTSS (Table 29). Almost all of them agreed that NOTSS is a valuable tool for reflective practice (96%) and a large majority saw its value as an adjunct to surgical skill assessments (81%). A large majority of nurses perceived that they were able to easily assess interpersonal and cognitive skills (92% and 85% respectively). This was noticeably different from the anaesthetists where the proportion was 60% and 27% respectively for being easily able to assess interpersonal and cognitive skills. Sixty-five per cent of nurses felt that NOTSS would enhance patient safety. However, four scrub nurses felt that NOTSS assessments added too much time to their lists.

Assessment ratings for assessment tools

In order to make it possible to appreciate the results, we have summarised the structure and rating profile of each tool.

Procedure-based assessment rating profile

The PBA instrument consists of a checklist of up to 62 items, each of which may be scored as ‘performed to a satisfactory standard for CCT’, ‘development required’, or ‘not observed/not appropriate’. The items are clustered into the domains of ‘consent’, ‘preoperative planning’, ‘preoperative preparation’, ‘exposure and closure’, ‘intraoperative technique’ and ‘postoperative management’. Each of the 15 index procedures has a different number and variety of task-specific items, so comparison at the item level is not possible. The structured checklist is therefore presented as the adjusted total item score (ATIS). This is a proportion of the perfect score of 1, based on the mean of the completed/appropriate items by converting satisfactory to 1 and development required to 0. In addition, assessors make a summary global assessment of progression towards independent practice. This is presented as the level, whereby level 1 is unable to perform the procedure, or unable to perform that part of the procedure that was observed, under supervision; level 2 is able to perform the procedure under supervision; level 3 is able to perform the procedure with minimum supervision (needed occasional help) and level 4 is competent to perform the procedure unsupervised (could deal with any complications that arose).

Objective Structured Assessment of Technical Skills rating profile

The OSATS instrument consists of three parts. Part 1 contains a checklist of up to 17 tasks unique to each index procedure. Each task may be scored as ‘performed independently’ (1), ‘needs help’ (0) or ‘not applicable’. The checklist is presented as the adjusted total task score (ATTS) – calculated in exactly the same way as the ATIS for PBA as a proportion of 1. Part 2 is a generic assessment with eight global dimensions of performance:

1. ‘respect for tissue’

2. ‘time, motion and flow of operation and forward planning’

3. ‘knowledge and handling of instruments’

4. ‘suturing and knotting skills as appropriate for the procedure’

5. ‘technical use of assistants’

6. ‘relations with patient and the surgical team’

7. ‘insight/attitude’

8. ‘documentation of procedures’.

Each generic skill may be assessed at one of three levels (0, 1, 2) based on behaviourally anchored descriptors of performance. The generic section is presented as the adjusted total generic score (ATGS), based on the mean of the completed elements out of a perfect score of 2. In Part 3 the assessor is required to provide a ‘pass/fail’ judgement by which the trainee has ‘achieved’ (1) or ‘failed to achieve’ (0) the OSATS competency. According to the guidance material for OSATSs, this judgement is to be based on a clear assessment algorithm that requires all checklist items (Part 1) to be scored ‘performed independently’ plus the general skill for ‘insight’ scored at the highest level of 2.

Non-technical Skills for Surgeons rating profile

The NOTSS instrument consists of four domains of performance, which are described in the NOTSS tool as categories. These constitute the skill categories: ‘situation awareness’, ‘decision-making’, ‘communication/teamwork’ and ‘leadership’. Within each category there are three element scores, which reflect behaviours relevant to the category. There are clear positive and negative behavioural examples provided for each category as these are designed to assess separate performance domains. There is also one overall category score which is an overall expert judgement based upon balancing the three element scores. Each category and element may be scored as ‘poor’ (1), ‘marginal’ (2), ‘acceptable’ (3) or ‘good’ (4). As the instrument is clearly intending to measure four separate domains, each NOTSS assessment is presented as four separate category scores: ‘situation awareness domain score’ (SDS), ‘decision-making domain score’ (DDS), ‘communication/teamwork domain score’ (CDS), and ‘leadership domain score’ (LDS). For the purposes of this analysis a ‘global score’ (GS) was also calculated as the mean of the four category scores to provide a G coefficient for the tool as a whole.

These 10 possible assessment ratings across the three tools are presented as a function of procedure and assessor type in Table 30.

Reliability of procedure-based assessment

Tables 31 and 32 display the variance component analyses for the ATIS and the level score respectively. For both scores, the ‘overall’ ability of the trainee (as judged across all procedures, cases and assessors) makes the largest contribution to any given score (variance 33% and 36% respectively).

Of the two ratings, the level score appears to provide a more reliable indicator of the trainee’s performance than the ATIS because it highlights the trainee’s aptitude for a particular procedure (variance 21% vs 13%), brings assessors of different stringencies into line (variance 3% vs 15%), and reduces assessor subjectivity (variance 16% vs 21%). Using either rating, trainees vary in their performance from case to case (variance 11% vs 9%) and the index procedures appear to differ in their difficulty (variance 11% vs 7%).

Table 33 shows the D-study table for the ATIS, comparing trainees on the same procedure and on a mix of procedures. Reliability for the same procedure (G > 0.8) can be achieved using four cases with one assessor per case, which equates to four individual expert judgements. Reliability on a mix of procedures can be achieved only with large numbers of cases and assessors.

Table 34 shows the D-study table for the level score, comparing trainees on the same procedure and on a mix of procedures. Reliability for the same procedure (using G > 0.8) can be achieved using three cases with one assessor per case, which equates to three individual expert judgements. Reliability on a mix of procedures can be achieved only with large numbers of cases and assessors.

Reliability of Objective Structured Assessment of Technical Skills

Tables 35 and 36 display the relative contributions that a number of key factors make to score variation for the ATTS and ATGS respectively. The pass/fail judgement was completed too infrequently by the supervising consultant (50% of cases) to allow useful variance component analysis. For both the ATTS and ATGS, the largest contribution to any given score across all procedures, cases and assessors is the overall ability of the trainee (variance 34% and 25% respectively). However, the influence of the other factors is very different for the two ratings.

Of the two, the ATGS appears to be a more reliable assessment because it highlights the trainee’s aptitude for a particular procedure (variance 7% vs 0%) and reduces the extent to which case-to-case variation influences the score (variance 8% vs 26%). However, assessor stringency has a greater impact on the ATGS than on the ATTS (variance 14% vs 9%), and both are much higher than the 3% for the PBA level score. Both the ATTS and ATGS attract a significant score variance, which is attributable to assessor case subjectivity (variance 23% and 21%), although this is similar to the PBA ATIS and the level score (variance 21% and 16%).

The striking difference compared with the PBA scores is the large element of assessor stringency variation that appears to be attributable to the assessor’s designation within OSATS scores (variance 4% and 16% compared with 0%). This reflects the differences in scores that are attributable to our two main assessor groups (clinical supervisors and independent assessors), whereby these assessor groups differ more in their stringency when using OSATS and agree more when using PBAs.

The ATTSs show 0% variance owing to both trainee procedure aptitude and assessor subjectivity over procedure. These are the two additional procedure-specific components used in the generalisability theory model for mix of procedures analysis. Therefore, the reliability of the ATTS, whether modelled for the same procedure or for a mix of procedures, generates the same D-study table estimates. For the ATTS, reliability (G > 0.8) can be achieved using five cases with one assessor per case (Table 37).

Table 38 shows the D-study table for the ATGS, comparing trainees on the same procedure and on a mix of procedures. Reliability for the same procedure (G > 0.8) can be achieved using five cases with one assessor per case. Reliability on a mix of procedures can be achieved only with large numbers of cases and assessors.

The ATTS and ATGS for the same procedure require the same number of cases and assessors per case (equivalent to five expert judgements) to achieve G > 0.8. The ATGS shows better overall reliability for assessing within each procedure, considering both the variance component analysis results and the slightly higher G-values within the D-study tables (0.83 vs 0.81). However, as the ATTS fails to reflect procedural variance, it would appear to be more reliable for making comparisons across procedures.

Reliability of Non-technical Skills for Surgeons

Table 39 displays the relative contributions that a number of key factors make to score variation for the NOTSS GS. The ‘overall’ ability of the trainee (as judged across all procedures, cases and assessors) makes the biggest contribution (variance 31%), but only by a small margin. The reason for the small margin is the fact that assessor stringency variation and assessor subjectivity strongly influence the GS (variance 27% and 20% respectively). NOTSS GSs show more variance owing to assessor stringency than either PBA or OSATS scores, although variance arising from assessor subjectivity is similar across all the tools (variance range 16%–23%).

The specific index procedure being observed has a much smaller influence on the score in the context of NOTSS-based non-technical skills assessment than it does in PBA-based surgical skills assessment (variance 3% compared with 11% for procedure difficulty and 5% compared with 21% for procedure aptitude). Baseline trainee case-to-case variation is similar to PBA ratings (variance 10% compared with 9% and 11%).

Table 40 summarises the factors that influence the scores across the four category ratings. Differences exist between the category ratings, some demonstrating greater or lesser proportions of true variance and case-to-case variation. The most striking observation is that, as with the GS, they are all highly influenced by assessor stringency and subjectivity, while demonstrating relatively procedure-independent scores.

Table 41 shows the D-study tables for the NOTSS GS, comparing trainees on the same procedure and on a mix of procedures. Because of the strong influence of assessor stringency and subjectivity, the reliability of NOTSS is lower than that for PBA. Reliability for the same procedure (G > 0.8) can be achieved using six cases and one assessor per case, i.e. six individual assessor judgements.

As procedure-specific factors exert a lesser influence on NOTSS scores than on either PBA or OSATS scores, the reliability of NOTSS scores on a mix of procedures is more similar to the reliability for the same procedure. Reliability between procedures (G > 0.8) can be achieved using eight cases and one assessor per case, i.e. eight assessor judgements.

Hypothesis 6: Effect of assessment on performance

Assessment (plus or minus video-recording equipment) will not affect the performance of trainees.

Trainees’ and clinical supervisors’ perspectives were sought on whether surgical performance was affected by the assessment conditions from case 96 onwards, representing a total of 342 cases. Both trainees and clinical supervisors were asked for their judgement (‘yes’ or ‘no’) as to whether the trainee performed differently for each separate case that was assessed within the study. Some clinical supervisors were unable to comment (n = 50) if they had not previously supervised a trainee either operating or performing a particular index procedure. Similarly, some trainees felt unable to comment (n = 14) for corresponding reasons. The anaesthetists, nurses and SCPs were not asked, therefore only PBAs and OSATSs are considered.

Table 49 displays the responses. Trainees felt that their performance was affected in 70 (20%) of the cases where they were able to give a response. Clinical supervisors felt that performance was affected in 41 (12%) of cases where they were able to give a response. In total, there were 92 cases in which either trainees or supervisors judged that performance was affected. This represents 27% of cases where the question was asked.

Owing to the complex hierarchical nature of the data (interdependent variables), there is no feasible statistical approach for testing this hypothesis robustly on the basis of actual score differences. However, we do provide the following post hoc raw comparison of scores: (a) concerned that assessment has affected performance versus unconcerned that assessment has affected performance (subjective); and (b) video-recorded versus not video-recorded (objective). We do not offer these comparisons to answer hypothesis 6, rather to raise new hypotheses for further work.

Table 50 compares the mean scores given on those cases where either the trainee or clinical supervisor responded ‘yes’ with the mean score of the remaining cases. Cases in which either the trainee or supervisor responded ‘yes’ received lower scores on every measure. These differences reach statistical significance for all but the ATTS and ATGS, even after applying the Bonferroni correction [p < 0.006 (0.05/8)].

A total of 120 cases were video recorded during the study, representing 27% of all recruited cases (n = 437). Considering the cases where trainee and trainer opinions were sought on the effect of assessment on performance (n = 342), 91 (26.6%) of these cases were video recorded.

Table 51 compares the mean scores given on those cases that were video recorded with the mean score of those that were not. Cases that were video recorded received similar scores to cases that were not. None of the small differences in the table reached statistical significance after applying the Bonferroni correction [p < 0.006 (0.05/8)].

A significant minority (20% of trainees and 12% of supervisors) perceived that assessment affected their performance. Interestingly, trainees who felt that observation affected their performance (subjective) performed less well than trainees who did not feel affected. However, trainees who were video recorded (objective) did not perform less well than trainees who were not video recorded. This post hoc analysis cannot answer hypothesis 6, but it does raise the interesting hypothesis that observation or video recording do not affect actual performance, rather a poor performance may be attributed to assessment or video recording.