A methodological study to compare survey-based and observation-based evaluations of organisational and safety cultures and then compare both approaches with markers of the quality of care

Structured questionnaire research instruments

Structured questionnaire research instruments typically adopt a typological or a dimensional approach, but vary in terms of their theoretical or conceptual underpinnings, their scope and their depth. 63 Several organisational climate and culture measures have been applied in health-care settings, for example:

• Harrison’s85 Organizational Ideology Questionnaire assesses the ‘ideology’ of the organisation and its relationship to the interests of its members and to the external environment. The tool has been used in the UK to examine of the effect of NHS reforms and the monitoring of culture over time. 86

• The Competing Values Framework (CVF)87–89 (originated and is better known outside health care) uses a typology of four dominant culture types based on core values that characterise an organisation, arranged on two axes: internal–external focus and flexibility/individuality–stability/control. It has a strong theoretical basis and has been used in UK health care. 16,90

• The Organizational Culture Inventory91 assesses 12 sets of normative beliefs related to 12 different cultural styles; these reduce to three general types of culture: constructive, passive–defensive and aggressive–defensive. The tool has been widely used including in health care92,93 but is too lengthy to be completed by busy clinical staff.

• The Quality Improvement Implementation Survey94 (QIIS) uses four culture concepts associated with the CVF and adds an extra dimension (‘rewards’).

As interest in health care and health services research in the 1990s and early 2000s shifted towards patient safety, attention turned towards developing measures of those aspects of organisational culture and climate that related to safety. In the USA, several high-profile safety climate and culture measures were developed and validated for use in audit and research in health-care settings. Pronovost and Sexton95 provided a useful review and guidance. Well-developed research instruments include:

• multiple versions of the SAQ, including the Teamwork and Safety Climate Survey25,96 and the Institute for Health Care Improvement Safety Climate Survey,24,96,97 all of which were developed and extensively tested by the University of Texas Center of Excellence for Patient Safety Research and Practice

• Nieva and Sorra’s Hospital Survey on Patient Safety Culture for the United States Agency for Healthcare Research and Quality (AHRQ)55,98–100

• the Patient Safety Climate in Healthcare Organizations questionnaire (PSCHOQ) (Stanford University with the Palo Alto Veterans’ Affairs Health Care System), also funded by AHRQ under its Systems Related Best Practices initiative,20 and based on several existing surveys including the Operating Room Management Attitudes Questionnaire,101 an earlier iteration of the SAQ; the PSCHOQ has been extensively used37,59,102–106

• Gershon and colleagues’107 Hospital Safety Climate Questionnaire which, in contrast with the more generic tools above, focuses on universal precautions for blood-borne pathogens; Turnberg and Daniell108 adapted it for use in respiratory care.

Each of these instruments takes a dimensional approach to evaluating safety culture by aggregating individuals’ levels of agreement with statements on Likert scales. The dimensions of safety culture that are explored include teamwork climate, safety climate, job satisfaction, stress recognition, perceptions of management, working conditions, absence of barriers to safe working practices, minimal conflict and good communication among staff, safety-related feedback, and many more.

In the UK, there are three notable strands of development of instruments drawing together organisational and safety climate:

• The MaPSaF is based on Westrum’s theory of organisational safety (reprised in Westrum, 2004109). It is a typological tool initially developed for use in primary care trusts (PCTs) to assess safety culture maturity. It is intended primarily for team-based reflection and development82 and has been tailored for use in acute, ambulance and mental health settings. 84

• The NPSA sponsored the development of organisational and safety culture instruments for use across the NHS. 110 Aston University has developed the TCAM (Team Climate Assessment Measure). 111 The dimensions addressed include task reflexivity, team stability, leadership, participative trust and safety, open exchange and interprofessional exchange.

• The Healthcare Commission (subsequently Care Quality Commission) National Staff survey was derived from a model developed by Aston University that links work context (including organisational climate) to the management of people, psychological consequences for staff, staff behaviour and experiences, and errors and near-misses that ultimately affect patient care. 112

Elsewhere in Europe, Silva and colleagues38 developed an organisational and safety climate inventory, based on the CVF,87 while, during the course of this study, the development of other instruments has begun. 113–117

Ethnographic methods

Because culture is a fusion of values, attitudes, perceptions, competencies and behaviours, it is very difficult to measure. However, evaluations of culture can be obtained from temporary immersion in the environment: either prolonged engagement, as in traditional ethnography,62 or shorter strategic immersion, as employed in clinical governance review and inspection. 32 Ethnographies provide rich, multilevel understandings of researched environments, but they are time-consuming, and comparison of multiple environments is difficult. A wide range of safety culture studies have adopted the ethnographic approach. 34,45,118–125

Strategic immersion in the environment

A major advantage of direct observation, in contrast with the questionnaire response, is that it enables researchers to see what people do and say rather than just what they say they do. 126 It can uncover how complex jobs are routinised together with ‘the tacit skills, the decision rules, the complexities and the discretion’ utilised in routine and marginal work (Smith,127 p. 221). Although traditional ethnography requires lengthy immersion in the researched environment by participant observers, accomplishing high-quality observational research during relatively brief periods of immersion is potentially achievable providing researchers restrict their studies to a topic or ‘lens’ through which to view the group they are studying. Willis128 (p. 557) refers to ‘focused (limited gaze) ethnography’ and ‘rapid (quick time) ethnography’, where speed is a virtue and a necessity. Strategic immersion represents a more structured approach to observation and supplementary data collection, to provide defined coverage at reasonable expense and to permit comparison between different organisations. For example, studies of nursing care have used checklists of features of good practice to structure strategic immersion,129–131 whereas observations to develop or evaluate quality improvement interventions are structured by the focus of the intervention itself. 33,118

Strategic immersion is unsuitable for settings or issues for which there is inadequate earlier work to define a framework for observation. However, previous work in the field of organisational and safety culture renders evaluation through strategic immersion possible in this study. In particular, the observation strand of this study benefited from an earlier 3-year ethnographic study in delivery units, which three of the study team completed,34 and which linked experience of setting ethnographic observations alongside highly structured observations. 132

During strategic immersion to observe health-care teams at work there is limited or no observer participation in the observed environment, as the limited time period does not allow the observer time to negotiate an active role in the team being observed. The method may therefore be described as primarily that of ‘passive participation’. 133 However, passive observers may ask brief questions for instruction and clarification, although in health-care settings this may not always be possible when staff are busy. The scope for immersion may be limited by the difficulties associated with the qualifications required to enter into many of the roles of medical work. Smith127 (p. 227) argues that diversity of observational approaches results not from ‘methodological sloppiness’, but from ‘real constraints governing the conditions under which researchers can and cannot conduct qualitative field research’.

Hammersley134 (p. 26), discussing observation, argues ‘there are always multiple, non-contradictory, true accounts possible of any scene and we need to know the basis for particular selections’. Actions of individuals are observable, but the meaning attached to the physical actions is divorced from the actions themselves. Researchers may apply their own meanings to these actions, which may be different to the meanings of the actors. 135 The degree of inference and, therefore, potential for error is likely to be greater when observing some features of health-care practice than others.

Developing the organisational and safety climate questionnaire

We wished to use prevalidated questions and scales within the organisational and safety climate questionnaire for staff because the aims of the study (see Chapter 1, Aims and objectives) centred on comparing different approaches to evaluating culture rather than developing and validating new research instruments. A review was conducted at the beginning of the study (2007) to establish the availability and psychometric properties of organisational climate and safety climate data collection instruments. The instruments identified were examined more closely by reading the associated websites and manuals. We also discussed (in person or by e-mail) the development, reliability and availability of a range of climate instruments with people engaged in their initial or continuing development. A small number of climate instruments developed in languages other than English were considered, but translation and revalidation were not feasible within our study funding and timetable.

The final selection of instruments from which to draw questions to evaluate organisational climate and safety climates used the following criteria:

• availability of material describing the development and testing of the instrument, scores from psychometric testing and benchmarking data

• stability and reliability

• extensive prior use in England and

• cost (no fees were levied for use of the scales and questions selected for this research study).

The staff survey questionnaire (see Appendix 3) was developed by combining groups of questions from the NHS national staff survey175 and the Teamwork and Safety Climate Survey. 25 We received permission to use questions from the NHS national staff survey in any month except October. This was to minimise any possible confusion with the annual NHS staff survey, which is distributed each October. The data collection timetable for this study was adjusted to satisfy this restriction.

The study questionnaire contained four sections of questions from the NHS national staff survey which were intended to capture facets of organisational climate, namely:

1. Q6 a six-item scale evaluating perceptions of the organisation, which in subsequent analyses we have termed ‘organisation’

2. Q7 two questions about whistle-blowing

3. Q8 a question about reporting errors, near-misses and incidents

4. Q9 a seven-item scale concerning organisational responses to errors, near-misses and incidents, which we have termed ‘error wisdom’ in subsequent analyses.

At research sites 1 and 2, the staff survey contained additional questions relating to other aspects of organisational culture, but response rates from these sites that were lower than we had hoped for prompted shortening of the questionnaire (see Sampling, delivery and maximising returns).

A further five sections of the questionnaire were intended to capture perceptions of safety climate and team factors or local work environment factors that are thought to be related to safety climate (see Chapter 2), namely:

1. Q2a–c a three-item scale, which we have labelled ‘overload’

2. Q2d a five-point Likert-scale item called ‘Relationships at work are strained’

3. Q3 a five-item scale evaluating supportive behaviours from the respondent’s immediate manager, which in subsequent analyses we have termed ‘line management’

4. Q4 one question each concerning working closely with other team members and meeting regularly to discuss effectiveness and improvement

5. Q10 teamwork climate scale (six items)

6. Q11 safety climate scale (seven items).

Questions 2–9 were reproduced from the NHS national staff survey,175 whereas questions 10 and 11 were Teamwork Climate and Safety Climate scales. 25 A minimal change to question 10 was required to reflect professional titles in study sites: in the questionnaires distributed to DU staff the word ‘nurse’ was replaced with ‘midwife’.

One question (Q5, Appendix 3) asked about trust-provided or trust-supported training, learning or development during the past year. This was included because CPD of individuals and teams is argued to be important for developing and maintaining positive organisational and safety cultures67 and to maintain job satisfaction and motivation. 81 Information on common forms of CPD was elicited in the form of closed questions (see Table 9) and space provided for additional responses. The listed categories of CPD overlapped, but all were included because previous research has shown that people find it surprisingly hard to call to mind examples of workplace or work-related learning. 176 Providing a variety of prompts, including the most popular approaches to CPD, was intended to help respondents recall CPD and to encourage them to recognise learning within a wide variety of activities.

Demographic and role-related questions were also included to permit analysis of whether these factors mediated responses to the organisational climate and safety climate questions and to address objective 8 (see Chapter 1, Aims and objectives). This study used response categories from the NHS national staff survey to maximise opportunities for comparison between the study results and trust-wide or national results from the same survey questions.

Piloting with clinicians associated with the university where most members of the study team were based established the usability and face validity of the questionnaire for clinicians from different professions working in DUs or EDs, and that it took < 30 minutes to complete.

Sampling, delivery and maximising returns

Clinical leads at the research sites were asked to identify staff who should be invited to complete the staff survey. Thus, midwives, nurses, doctors, support workers, administrators and managers (and, where appropriate, allied health professionals) were invited to participate in the survey, the exact selection being determined by team leaders’ definitions of team membership. However, clinical leads were asked to exclude certain groups whom we felt they would have insufficient consistent experience of the culture of the particular clinical area: students and members of staff who had joined the department < 4 weeks before the survey was distributed, and in particular Foundation Programme doctors (doctors in training in their first 2 years after graduation from medical school). Comparison of the national rotation schedules for these junior doctors and the data collection timetable for the sequentially researched study sites revealed that these junior doctors would be eligible for inclusion at some sites but not at others due to very recent rotations: we therefore asked that they be excluded from staff lists at all sites. It is a moot point whether these exclusions were necessary and advisable, as newcomers may evaluate the culture of their work environment very quickly and may be better able than more established members of clinical teams to identify safety concerns. Further research into the differing perspectives of newcomers and established members of staff may be useful.

Seeking a locally defined definition of the ED or DU team was part of the commitment stated in objective 1 to work with staff in participating trusts, respecting their organisational and professional knowledge. The definition of the departmental team was thus conceptually coherent for senior clinicians at each research site but differed between research sites. We accept that our design in this respect is vulnerable to the criticism that data between sites were not directly comparable. However, health-care teams do vary between departments and hospitals, according to local need, resources and preferences; so strict comparability between staff survey samples from multiple research sites would be an unrealistic ideal for any study. In this study it was deemed more important to examine how team compositions and perceptions of the team varied. For example, some sites had dedicated allied health professionals, administrative and clerical staff, porters and cleaners, and one (an ED at two geographically separated locations) had dedicated ambulance staff; others had few or none of these. The use of agency and/or locum staff for long periods (months or even years) varied substantially between sites: these may be key team members but do not appear on trust staff lists. We would suggest that the variability we have documented is a feature of all multisite studies of teams, but rarely made explicit.

Hand delivery and collection of questionnaires at 16 research sites was not feasible for research fellows, and preliminary work with research sites established that local clerical or clinical staff would not be able to undertake this work, so a postal survey was conducted. Questionnaires were sent to all designated members of staff along with a covering letter and a project information sheet. A stamped, addressed envelope was provided for the return of the questionnaire. One reminder was sent to non-respondents after 2 weeks. A unique identifier, which could be removed by respondents if desired, was placed on each questionnaire to prevent unnecessary reminders. The key to the identifiers was stored separately from the returned questionnaires and destroyed when no longer required. Each distribution of the questionnaire required minor updating to reflect the specific clinical area, contact details for queries and requested return dates. The data collection timetable of sequentially researched sites involved a risk of late returns from one site being allocated to the next site. To avoid this, we alternated the named contact for questionnaire responses at the first 10 sites. This strategy was insufficient at research sites 11–16, where data collection overlapped to compensate for earlier delays in the data collection timetable. For these sites different coloured paper was used for the questionnaires to prevent misallocation of responses.

Because most staff surveyed were nurses or midwives, we anticipated that the response rate would be no higher than 50–55%,30 even with the inclusion of elements that are known to support better response rates in the general population: sending from a university address, including a stamped return envelope, sending a reminder and offering an incentive for returning the questionnaire177 (entry into a prize draw with the opportunity to win one of 10 gift vouchers for a popular high-street store to the value £10). Salience is an important factor influencing response rates from health-care practitioners and the general population. 29,177 Patient safety was expected to be a salient issue in DUs and EDs, since this study coincided with a peak of safety-focused policy and practice development initiatives. However, it is possible that, particularly by the time of data collection at the later research sites, staff may have begun to be jaded by multiple waves of safety-focused activity. Response rates were lower than hoped for at the first two research sites (20% and 27%; see Table 5), and two steps were taken with a view to improving response rates. First, the questionnaire was shortened,177 largely by omitting questions about matters with a less immediate relationship to patient safety culture than those that remained (e.g. work–life balance; physical attacks on staff). In the final version in Appendix 3, 52 separate responses are requested, whereas the original questionnaire requested 107 separate responses. Secondly, changes were made to the use of the prize draw as an incentive for staff to complete and return the questionnaire promptly. Initially, the draw (for 10 gift vouchers) was made after 2 weeks but, to encourage responses to the reminder letters, a draw for four of the prizes was held 2 weeks after the second mailing at site 3 onwards. The unique identifiers were used to trace prize winners.

Response rates were variable and remained lower than hoped for, although similar to comparable studies. 100,178 A final attempt to increase response rates was trialled for research sites 11–14. This will be discussed in Chapter 4, Examining influences on response rates.

Analysis

Data collection

Six 1-day visits were undertaken at each hospital by at least two observers, which allowed sustained non-participant observation to see how staff behaved over a number of hours, and sampled sufficient time points to gauge the range of work and activity levels. In recognition of fluctuating workloads and staffing levels, the visits were distributed throughout the week. Data were, therefore, collected at all periods of the day excluding midnight to 6 am, and on all days except Sunday.

Researchers recorded their observations in hand-written field notes. These were made during observation periods, then reviewed and annotated within 24 hours. Field notes contained a minimum of identifying information, for example labelling participants in any documented interactions by their profession and level or role rather than by name or initials. No identifiable patient information was recorded, although patients were anonymously included in notes if their interactions with staff were of relevance. See Developing an observation prompt list for description of developing prompts to guide data collection.

Observers positioned themselves near the functional centre of the department, either the whiteboard where bed occupancy and the progress of care are recorded or the desk or counter where computer monitors were sited (these are often, but not always, colocated). The distributed nature of ED work46 (Rationale for focusing on delivery units and emergency units) necessitated the selection of one functional centre within EDs: ‘major injuries’. Observers interacted with staff (provided this did not mean interrupting their work) if observers wanted information to help interpret what they observed or staff wanted to talk to us about our research or their work. Information sheets were provided for use as leaflets or posters or both, and we offered to attend briefing sessions for staff, as well as pre-meetings with local clinical leads. All of these information sources pointed out that staff could ask observers to leave the vicinity, or themselves move more than 3 metres away, if they wished what they were saying or doing to be excluded from our data. But observers noted events and conversations that were clearly visible or audible to anyone, including patients and visitors, in the vicinity, regardless of distance.

Observation data were supplemented by an orientation meeting with a senior nurse or midwife at each site. This enabled observers to contextualise and interpret what they observed, and included questions about safety-related matters that we were unlikely to be able to answer from observation alone: staffing establishment, maintenance of equipment, and updating of protocols. The timing of such meetings varied, depending on the availability of staff; some took place before observations, some afterwards. We believe that such variation did not significantly affect our observations: when orientation interviews occurred later rather than earlier, we tried wherever possible to ask staff during observations about factors that seemed directly relevant to what we were observing; when the orientation meeting took place earlier rather than later, we sometimes contacted the senior nurse or midwife concerned at a later date by telephone or e-mail to ask questions arising later. Observers met between observation sessions to debrief and exchange information and observations: these meetings served also to ensure that each observer was keeping all the items on the topic guide in mind.

Within EDs observation was planned within major injuries and minor injuries sections but not in the resuscitation and paediatric sections. This decision was mainly guided by the additional ethical concerns raised by observing in the latter areas and a strong desire among the researchers not to risk distracting staff in the normal conduct of their work, which seemed more likely in these areas.

At some sites the project research fellows were joined by a service user or an experienced service provider who did not work at the research site (see Chapter 6, Data collection summary). The purpose of this was to ensure that the perspectives and interpretations of the research fellows were supplemented and if necessary challenged by clinical and lay viewpoints. Service provider coresearchers were able to offer contextual and explanatory information, whereas the role of the service user coresearcher was to ensure that the research fellows were not too closely aligned with clinical perspectives at the cost of critical distance. These coresearchers received training from one of the project research fellows prior to visiting any of the research sites. They observed from the same vantage points as the project research fellows, and using the same prompt list, though not at the same time. They also made hand-written field notes, which were subsequently passed to the research fellows to be included in data analysis. Each coresearcher met with the research fellows to discuss and elaborate upon field notes and subsequent reflections.

Each site was offered early feedback on data collected in the three strands, A–C. Where this opportunity was taken up, discussion of the observation occasionally corrected minor misunderstandings or added new material. Field notes made during and immediately after the feedback meetings also became part of the strand B data set.

Developing the observation prompt list

Mindful that the commissioning brief and study protocol (see Appendices 1 and 2) required quantified observations, the research team initially planned to develop a checklist that would be scored in situ. A provisional detailed checklist was developed to pilot interview and observation data collection focused on indicators of organisational and safety culture drawn from our previous work34 and published literature (see Chapter 2). It included some of the factors that influence safe clinical practice developed by Vincent and colleagues:56 organisational and management factors, work environment, staffing levels and skills mix, team factors, individual (staff) factors and task factors. The aim was to identify a sufficient but manageable range of items to support evaluations of organisational and safety cultures. The provisional checklist was piloted in an ED that is not included in the main study and it was anticipated that the checklist would undergo further refinement as the study progressed.

Provisionally, it was planned that items in the final checklist would be rated by the researchers who had conducted observations and supplementary interviews, using the following ordinal scale, which would permit numerical scoring: C, evidence that this facet of organisational or safety culture is consistently in place; P, partially in place/dependent on individuals; N, not in place. There was provision for rating DK, don’t know/can’t assess, which needed to be used sparingly to prevent missing scores from undermining the overall scoring. Space was provided in pilot checklists to note items of evidence to support the rating given. While piloting the checklist, researchers (experienced ethnographers) also wrote extensive field notes to inform and evidence their provisional ratings. Rating of sites was undertaken provisionally in the field and final ratings for each item allocated on the return from the field, based on the two researchers’ provisional ratings and discussion and comparison of field notes and observations. The use of field notes permitted the accumulation of evidence to inform ratings, and permitted researchers to cross-reference with one another’s observations, offering the opportunity to check inter-observer reliability. The initial checklist for the study was always anticipated to be too wide-ranging for real-time use in clinical areas, but the items continued to be useful prompts for researchers reviewing the comprehensiveness and comparability of the observation and interview data gathered at each research site so it was retained as a prompt list (see Appendix 5).

During piloting researchers found that some items in the initial checklist overlapped and it was difficult to gather evidence relating to some items. Researchers were sometimes uncertain about assigning ratings quickly in situ, and preferred recourse to detailed field notes. They were also concerned to make provision for recording unanticipated facets of the local environment or workplace practices that influenced workplace and safety cultures. Consequently, semistructured field notes were made during observation periods and annotated soon afterwards. The field notes were then coded with reference to the prompt list, which later evolved into the scoring guide for holistic assessment of culture (see Appendix 6).

While all items on the detailed prompt list (see Appendix 5) were useful to sensitise observers to relevant features of the clinical setting, and to structure field notes, assigning ratings for every item was not always possible, even with discussion between observers and reference to field notes. Overlap remained a problem, so the decision was taken to collapse items into fewer higher-level categories that could be scored consistently (see Appendix 6). We will discuss the scoring of the instrument in Appendix 6 in Chapter 4, quantifying semistructured observations.

Analysis

Extensive field notes, guided by sensitising prompts in Appendix 5, formed an assessment of culture in each of the research sites. (A separate inductive qualitative analysis of the field notes will be possible but does not form part of the commissioned study reported here.) Quantification of the results was required in order to perform the comparisons intrinsic to this study and its commissioning specification. To achieve this the observations were categorised using eight domains (see Appendix 6), each of which was scored separately following the period of strategic immersion. Three of the categories related to safety culture at organisational level, and five at team level. Two research fellows (SA and MR) scored each site separately, after reading all field notes (from preliminary meetings, observations, brief conversations during observation periods, coresearchers and feedback meetings). They then met to agree a final score.

The organisation and work environment factors scores reported in Chapter 6 (see Table 27) capture safety-related facets of the organisational culture in each of the research sites, namely the adequacy of staffing and premises, the availability of sufficient well-functioning equipment, and support from managers, administrative staff and other support roles. The mean of these scores was calculated to form a summary observation-based organisational culture score for use in subsequent analyses.

The team factors scores in Table 27 (undertaking informal training and supervision, particularly of juniors and students; leadership offered by senior clinicians, team members taking responsibility for their own work, levels of individual and team situational awareness; evidence of respect, warmth and collegiality within communication and actions; the quality of information exchange within the team and evidence of mutual support within and between professions) capture several facets of the local culture of teamwork that have been linked to safety culture (see Chapter 2). The mean of team factors scores became a summary, observation-based teamwork culture score.

The grand totals in Table 27 include organisational, work environment and teamwork factors that relate to safety. These form a more holistic evaluation of safety culture and the mean of the scores for all eight items in Table 27 are the observation-based holistic safety climate score for each research site. However, this assessment is not distinct from the observation-based organisational culture assessment comprising organisation and work environment factors. Consequently, it was not possible to test hypothesis 2b as written. Instead, the observation-based assessments of organisational culture and teamwork culture are compared in Chapter 6.

Audit data collection

At each hospital we aimed to audit 50 consecutive cases for each condition, working backwards for up to 6 months from the date the researchers first became noticeable in the clinical areas by commencing observations. The audit worked backwards to avoid any concerns about a possible Hawthorne effect emerging after researchers were encountered in clinical environments, leading to more than usually diligent record-keeping. The time limit of no more than 6 months was set so that the audit data could be regarded as contemporaneous with the observation data and staff survey responses. For MSL the audited markers spanned mothers’ and babies’ clinical records. Mothers’ records had to be audited first to obtain required audit data and identify the hospital record number for each associated baby, then the babies’ notes could be ordered and audited. Data extraction sheets were developed to structure the audits. One example, the data extraction sheet for FNoF, is provided in Appendix 7.

Non-clinical auditors were preferred, as earlier research found this reduced hindsight bias,186 and more recent research has found that non-clinical auditors are no less reliable than clinical. 145,156 The study aimed to recruit local audit clerks and provide study-specific training and ongoing support. The study budget made provision to reimburse trusts for the audit clerks’ time. Audit clerks sent data extraction sheets to the research fellows, who checked the extracted data and made enquiries about any anomalies they found (e.g. suspected duplicates, apparent inclusion of cases that did not meet inclusion criteria, suspected recording errors and patterns within missing data that suggested that certain sections or types of clinical record were being overlooked). Once enquiries were satisfactorily resolved, data from the extraction sheets were coded and data entry was checked before statistical analysis. In some sites, the research fellows carried out the audits (see Chapter 4, Engaging non-clinical auditors).

Aligned with objective 1 (see Chapter 1, Aims and objectives), which sought to work supportively with staff in the research sites and offer prompt feedback to allow local development, we undertook some finer-grained data extraction. In addition to the minimum data required to assess whether the audit standards had been met, we collected additional contextual data to enable richer feedback to research sites. For example, rather than record simply whether or not aspirin was given immediately, we also recorded how long it took to give aspirin in non-immediate cases. Thus, research sites could be given information about how far short of the standard they fell during the sampled period.

Analysis

In order to calculate each site’s scores in relation to each standard, we calculated the number of cases where the standard was met as a percentage of the total number of cases to which it applied (i.e. removing cases from the denominator such as, for standard ACS2, when aspirin had already been given by the ambulance service, or the patient had taken regular daily aspirin, or the patient was allergic to aspirin). We then calculated the mean level of compliance for each condition at each site so that, regardless of the number of audit standards per condition, each condition would contribute equally to the summary audit score for each site. It is appreciated that this method of calculating one summary score for each condition may result in a good overall score being obtained for a condition even when some of the individual standards had very low values. It was felt, however, that each standard should be given equal weighting within each condition. Possible pitfalls in this approach will be discussed in Chapter 9, Objectives. The mean scores were inspected for normality (see Chapter 3, Testing for normality) and then converted to the standard normal distribution. Finally, summary audit scores were calculated as the mean of the condition-specific z-scores for the three conditions measured in each research site (ED conditions or DU conditions, as appropriate).

Samples and response rates

Some clinical leads identified team membership themselves (see Chapter 3, Sampling, delivery and maximising returns); others delegated the task to junior colleagues or, most often, administrative staff. No site had a consolidated staff list, so different lists for different professional groups were provided. Generally, departments found lists of nurse or midwife names easy to provide, though some did not provide roles or bands, and some were out of date (excluding more recent arrivals whose names we learnt during observation: where possible, these were added). Lists of doctors proved more challenging (in one case, no list was provided at all, despite multiple requests, and we had to rely on the hospital website, which listed only consultants). These added to factors already identified (see Chapter 3, Sampling, delivery and maximising returns) meant that samples were not comparable across sites. In theory, we could have tried to gather further information in order to edit and harmonise lists, but we were convinced that we would have encountered considerable resistance from overstretched departments that would have been sufficient to prevent us achieving our objective.

It became evident that some databases were not quite up to date because 21 questionnaires were returned with a message saying that the individuals to whom they were addressed no longer worked in the department. These have been subtracted from the figures shown in the ‘surveys sent’ column of Table 5. We cannot know how many other questionnaires could not be delivered to the named recipients but were not returned. Turnover of staff may mean that a small number of new staff who could have participated in the survey were not invited to do so. This was occasionally apparent when observations for strand B identified a member of staff from an eligible staff group who was not listed for inclusion in the staff survey. Some of the lists we received contained excluded categories (see Chapter 3, Sampling, delivery and maximising returns), mainly Foundation Programme doctors. We tried to clean the lists before questionnaires were sent, within the limits of the information fields provided in each list. Many sites did not provide sufficient information to allow us to augment Table 5 with a breakdown of the surveys sent by professional group. However, where the professions of respondents are known they are summarised in Chapter 5, Demographics and role-related characteristics. Interactions with key contacts suggested that requests to improve lists were likely to be perceived as problematic, so we chose to accept what we were given, even when imperfect. This simply reflects the reality of conducting research in increasingly busy, pressurised health services: pragmatic decisions must be made or research would halt. 126 Nevertheless, the pragmatic decisions should be noted to allow reflection on their consequences. Published reflection on pragmatic decisions and known or likely consequences occurs less frequently than that which might be considered warranted by those who are familiar with the day-to-day imperfections and predicaments of research. This is likely to be attributable to publication bias and also, in Goffman’s terms,197 to a degree of ‘idealization’ in the ‘front stage performances’ of researchers.

Decisions about which groups of staff to include in the survey varied, for example site 3 was particularly interested in eliciting the perceptions of support staff and included a wider range of support roles than was the case at other sites. Because site 3 adopted a broad definition of team membership, we distributed twice as many questionnaires to staff at site 3 as at site 11 (see Table 5), although, as shown in Table 3, the two sites had a very similar patient throughput. Varying local decisions mean that there is no simple relationship between the size of the clinical department, as indicated by patient throughput, and the number of people designated team members. Furthermore, we would also expect the number of people designated team members to be affected by the local prevalence of part-time working, data which we do not have.

The overall response rate of 27.6% was very similar to the maximum response rate of 27% achieved by Catchpole and colleagues,178 who distributed safety climate questionnaires to staff in two surgical departments. It was a little higher than the 22% estimated response rate achieved by Sarac and colleagues,100 who distributed Hospital Patient Safety Culture (HPSC) questionnaires to clinical staff in seven acute hospitals in Scotland. However, the response rates for this study were consistently lower than the trust-wide response rates published for the NHS annual staff survey (Table 6). Comparing Tables 5 and 6, it is interesting to note that both data collection exercises yielded noticeably lower response rates at the trust in which research sites 7 and 8 were located. Ideally, we would have wished to conduct a non-response analysis to discover any systematic differences between survey respondents and non-respondents, based on sample characteristics provided by research sites with staff lists for the survey (e.g. profession, grade and demographic variables). However, as noted above, it was difficult for research sites to provide us with up-to-date staff lists and we did not obtain sample characteristics beyond those that routinely appeared on staff lists in each research site, mainly, but not always, profession, grade and gender. Missing and inconsistent information on sample characteristics precluded non-response analysis.

Examining influences on response rates

Multilevel modelling estimated the overall site within hospital response rate to be 29.5% [standard error (SE) 3.2%]. Variation in response rates was partitioned as follows:

• 60% at hospital level (estimated variance 0.006, SE 0.005, χ² = 1.820, 1 df, p = 0.177)

• 40% at research site level (estimated variance 0.005, SE 0.003, χ² = 4.000, 1 df, p = 0.046).

Variation in response rates associated with differences between hospitals was not statistically significant but variation due to differences between research sites was significant at the 5% level. Adding site-level characteristics to the model, as described in Chapter 3, Multilevel modelling, showed that the type of service (ED or DU) and the quartile in which the department lay (see Table 3) did not have significant effects of the response rates. There was a very small, statistically significant, effect for the number of questionnaires sent (estimated parameter –0.001, SE < 0.001, χ² = 4.806, 1 df, p = 0.028). This means that response rates are predicted to fall by 0.0032% (0.001 × SD of 3.2%) for every additional 62 (1 SD) people included in the staff survey. This effect is too small to have any practical effect linked to large departments or departments defining the team very broadly.

Adding the number of questionnaires sent as a fixed factor in the multilevel model reduced the residual site-level variance slightly to 0.004 (SE 0.002, χ² = 3.993, 1 df, p = 0.046). There remains significant site-level variation that cannot be explained by the site-level variables we collected in this study. The residual site-level variation may be partly due to site-level factors not recorded in this study. It may also be partly due to the characteristics of individuals working within the research sites. The analyses in Chapter 5 include individual-level factors.

In retrospect, it seems that the lower-than-hoped for response rates at sites 1 and 2 are not adequately explained by the length of the questionnaire, as response rates were lower at some subsequent sites where the shorter questionnaire was used. A further attempt to increase response rates was made by providing an online option at sites 11–14. An online survey was not offered at the beginning of the study because we knew that many front-line staff had only limited access to computers. However, web access using hand-held devices and wireless networks became popular during the course of this study and it was thought that respondents might find this more convenient and attractive. The online questionnaire was developed using Smart-Survey™ online survey software198 and hosted on the Smart-Survey website. However, we were still reliant on conventional post to alert staff to the survey and to provide the web address, as research sites were not asked to supply e-mail addresses for individual staff. Consequently, we could not provide the convenience of clicking on a hyperlink embedded in an e-mail. No-one at research sites 11–14 submitted an online survey. For this reason, it was not offered at sites 15 and 16.

Observable activities

Within EDs, the observations were piloted within ‘major injuries’ and ‘minor injuries’ areas, but proved unsatisfactory in minor injuries areas, where cases tended to be dealt with by individual staff in closed rooms or curtained cubicles. Observers at the record-keeping and workload management hub of minor injuries had insufficient opportunity to observe interactions and work practices that would allow scoring in the eight domains of the observation prompt list (see Appendix 5). The conditions that had been selected for the criterion-based audit (see Chapter 3, Audit data collection) were all treated within major injuries, making it logical to restrict observation to the major injuries sections of EDs.

There was considerable variety in what could be observed from each vantage point (see Table 7 for a summary of this). In the list that follows, ‘fully observable’ means that the specified activities were as a general rule both visible and audible from our vantage point. ‘Partly observable’ means that activities were either only visible, or only audible, or partly visible and/or partly audible.

• At eight research sites the whiteboard was fully observable; in four, partly observable; and in four (all EDs), there was no whiteboard (computer monitors were used instead, not observable).

• At four research sites (all DUs), medical and nursing handovers were fully observable; in 11 sites, some or all handovers were partly observable; in one site, handovers took place off the ward.

• At one research site the main area for doctors to meet, review care and write up clinical records was fully observable; in four sites it was partly observable; in 11 sites it was not observable.

• At six research sites the main area(s) for nurses to meet, review care and write up clinical records were fully observable; in nine sites, partly observable; in one site, not observable.

• In 14 research sites the shift coordinator’s base was fully observable; in two sites, partly observable.

• In eight research sites (all EDs), some interactions with patients were partly or fully observable; in eight sites (all DUs), none or almost none were.

These differences presented some challenges to observers, though it was possible to meet these by sensitisation to different sources of evidence. For example, where we could not observe handovers directly, we listened carefully to discussions about patients for evidence that staff were or were not sufficiently informed. Where computer screens had replaced whiteboards, we carefully observed how staff used them (e.g. we did not need to be able to read patient details to assess staff interactions and decision-making while looking at the screen). Furthermore, greater visibility was not always an advantage; in site 15, doctors worked primarily at the central hub, but the amount of activity meant that conversations though visible were often inaudible.

The organisation and work factors scores capture some facets of the organisational culture in each of the research sites. However, some important aspects of safety culture could not be observed, for example:

• the existence and effectiveness of appropriate systems for ensuring CPD

• the existence and effectiveness of appropriate systems for reporting and acting on errors and ‘near-misses’

• routine checking and maintenance of equipment (which is often periodic and infrequent)

• the availability of information to support evidence-based clinical decision-making where this was in computerised rather than in paper form.

Though in some sites, these might, by chance, be observed, or mentioned in passing by staff, we could not assume that no data reliably indicated an organisational deficit.

Recruiting service user and service provider coresearchers

In addition to the research fellows, other observers were recruited where possible: a service user observer, a midwife, a midwifery lecturer and an A&E lecturer-practitioner visited some sites. These coresearchers made observations on one day at each site they visited. Details of observers at each research site can be found in Chapter 6 (see Table 26), which shows that a service user observer could be present at only three research sites. This highlights that the intention of working with service user coresearchers at each research site was far from realised. At the first four sites, this was because we had not yet succeeded in recruiting service user coresearchers. We attempted to do this by approaching NHS service user panels in London and using the researchers’ personal networks of potential service user coresearchers. We explained the study and the funding available to compensate coresearchers for their time and expenses. Two potential service user coresearchers from an NHS service user panel expressed an interest in joining the study. However, one did not respond to communications after an initial meeting, while the bureaucratic process of securing research governance approvals separately for each of the study hospitals deterred the other, who withdrew.

Efforts to recruit service user coresearchers recommenced, and one (DP) was recruited before the data collection began at hospital 3 (research sites 5 and 6). Supported by the project research fellows, DP completed the (different) forms required by subsequent research sites and provided the required references. This allowed CRB screening, occupational health screening and any further local procedures antecedent to each hospital issuing permission for DP to observe. At some hospitals this process took many weeks and permission could not be secured before completion of that phase of data collection. Consequently, the service user co-observer was able to work only at research sites 6, 15 and 16.

Identifying potential service provider co-observers was straightforward, although their clinical and teaching duties often meant that they were unavailable to accompany the research fellows (sites preferred to be introduced to additional observers by researchers they already knew). Service provider co-observers worked at eight research sites: 1, 6, and 11–16. They also provided a clinical perspective during discussions of observations made by research fellows at all sites.

Quantifying semistructured observations

A number of ways of grading observations were trialled and discussed (see Chapter 9, Tensions and challenges in quantifying observational data). The scoring system around which consensus emerged is summarised in Appendix 6. The notes of observations were analysed and grouped in eight categories, and then the data within each category were scored as follows:

1. 0 consistent lack of features of a safety culture

2. 1 frequent lack of features of a safety culture

3. 2 frequent presence of features of a safety culture

4. 3 consistent presence of features of a safety culture.

It did not prove possible to make confident, consistent decisions with greater differentiation than this four-point scale.

Grouping observations into categories before scoring differed from the approach advocated by others,199,200 who emphasise the scoring of constituent items rather than overarching domains. Trials of the item-based approach did not work well in this study, which was seeking a quantifiable holistic assessment. Following independent trials of item-based scoring by research fellows, debates to achieve agreed final scores were protracted but did not yield results in which the researchers had confidence; at this point, the eight categories were decided.

Engaging non-clinical auditors

We intended to engage trust-based audit clerks (or equivalent non-clinical personnel) to extract anonymous data from patient records (see Chapter 3, Strand C: criterion-based assessment of quality of care), reimbursing the trust for associated costs. Pre-study negotiations indicated that staff members might be interested in carrying out the work as overtime. During the recruitment phase for each site the need for a local auditor was highlighted and discussed. One site, which declined to participate, felt that staff would not be available for this work. All recruited sites indicated confidence that local auditors could be identified to support the study: either within their contracted working hours (with reimbursement for the trust) or as paid overtime. For example, one trust used this work to extend the employment of an administrator whose previous project was finishing; another viewed the audit work as suitable ‘light duties’ for a staff member returning from a period of sickness. However, the confidence of other trust key contacts proved unfounded. The study coincided with a period of increasing financial pressures in health care: staff workloads were rising and vacant posts were being scrutinised with a view to cost saving. Some individuals whom senior staff expected to undertake this study’s audit work were unable to do so within contracted hours and unwilling to undertake audit work as paid overtime. Some could begin the audit work only after a substantial delay to complete other work, or conducted the audit work over an extended period at the margins of their main roles. Researchers negotiating the recruitment of study sites could anticipate problems with staffing audit work, ask pertinent questions and encourage contingency planning, but could not really know the local situation until the negotiation of data collection with nominated individuals.

At the first research site, a member of the administrative staff expressed an interest, though in the event was not available. At the second, the clinical and central administrative departments were unable to identify any staff for this role. Two research fellows (SA and MR, both non-clinical health services researchers) had already been awarded honorary contracts with the trust for the research, and were permitted to carry out the audit. The researchers found the audit process very complex and time-consuming owing to limitations in administrative systems (see Chapter 9, Limitations of administrative systems) but also highly informative. Consequently, they chose to continue conducting audits at the next six sites, after obtaining all necessary permissions and honorary contracts: they checked each other’s work, sensitised by the process of checking described in Chapter 3, Audit data collection. This direct engagement proved extremely valuable in informing the training and support of local audit clerks at later research sites. Furthermore, it can be argued that non-local auditors are ethically preferable as they are very unlikely to know patients whose notes they are scrutinising.

The study coincided with a period during which National Information Governance Board (NIGB) was established. One hospital required the study team to apply to the NIGB for a ruling on the acceptability of research fellows extracting data from clinical records. The NIGB ruled that individual patient consent would be required, unless members of the care team carried out the audit. It is a moot point whether trust-based audit clerks are members of the care team, in which case this NIGB ruling appears to rule out a wide range of quality improvement audit work currently conducted by non-clinical auditors employed by NHS trusts.

Seeking individual patient consent was impractical for this methodological study for a variety of design and resource reasons. We therefore asked the eight remaining research sites and associated trust administrative departments to identify non-clinical staff to extract anonymous audit data. Staff were not always able to begin work immediately or to devote much time to the work, and this led to delays of several months, necessitating an extension to this study (see Chapter 1, Funding for this study), and severely reducing the time available for analysing audit results and the comparisons that lie at the heart of this methodological study (see Chapter 8). Eventually, all but three of the remaining research sites identified non-clinical staff who could undertake data collection for this study, either within their normal working week or as paid overtime. Despite considerable efforts, research site 12 was unable to identify a non-clinical auditor and we accepted data extraction by a health-care assistant. At sites 13 and 14, no trust-based auditors could be found. The Research Office offered to issue the research fellows with honorary contracts in order that they could be regarded as trust staff, and thus allowed to conduct audits.

Each trust-based auditor was visited by a research fellow (SA) to discuss the data extraction required and varying local practicalities for identifying 50 consecutive cases for each condition, ordering clinical notes and identifying local conventions with respect to recording the data required by our audit (e.g. in some research sites a mixture of paper-based and electronic records had to be consulted). The use of the data extraction sheets was explained. SA provided ongoing support by telephone or e-mail until each audit was complete. This training and support helped to ensure the quality, completeness and consistency of audit data extraction by different auditors across multiple sites. It would have been difficult to provide if the research fellow had not had first-hand experience of auditing patient notes.

Cases audited

The study aimed to audit 400 cases for each selected condition: taking 50 consecutive cases from each emergency department or delivery unit (50 × 8 = 400), working backwards for up to 6 months from the date the researchers became noticeable to the whole clinical team by commencing observations. Inclusion and exclusion criteria for the audited cases were listed in Table 1. Table 8a and b shows the degree to which the audit target was achieved for each condition.

The main reason for shortfalls was human error (miscounting audited records) or the discovery during university-based data entry for analysis that the data for a particular case, which had been extracted anonymously from hospital records, were in some way too flawed to include. In a very small number of cases, the same record had been audited twice. As Table 8a shows, the audit of FNoF cases at site 1 failed to identify sufficient cases within the 6-month period (see Chapter 3, Audit data collection). The audited number of MSL cases at research site 14 fell short of the target because this small DU experienced only 16 cases that were eligible for inclusion in study during the relevant 6-month period and for which case notes could be found.

Case note data quality

Data relevant to our needs were frequently missing from the case records. Some records that were present were self-evidently unreliable (illegible, or inconsistent, particularly with respect to timings). Site staff at a number of sites told us that it is normal practice when auditing their own service to exclude notes in which relevant data are missing or unreliable. This is clearly appropriate where the object of the exercise is to begin the audit cycle of service improvement: inclusion of records with missing data would weaken the evidence base for service improvement. However, it is important not to lose sight of the proportion of case notes with missing data: an indicator of quality in its own right. Because ours was a comparative methodological study, and we were studying the validity and practicability of case note audit as a method, we included notes with missing data. The assumption that recording and activity correlate reasonably well is intrinsic to this method. Ideally, we would have included 50 cases for which data were complete while also noting details of incomplete cases: time and resources did not allow this, however.

At feedback sessions, clinicians often argued that unrecorded observations and interventions would, nevertheless, generally have been carried out. Such explanations have been offered for many years,201,202 and, though they may be true, they are hard to substantiate retrospectively.

Clinicians also pointed out that, in some cases, monitoring would be continuous, with only occasional notes being made of readings: for example, oxygen saturation may be measured continuously in asthma patients, most of which measurements would be unrecorded. Staff said also that some observations would be recorded only if problematic; for example, midwives at several DUs told us that they would record show and blood at initial assessment only if there were grounds for concern.

It also became clear during feedback that not all sites aimed to meet some standards. For example, at site 5, nurses do not prescribe to ACS and FNoF patients, and it is unlikely that patients will see a doctor ‘immediately’ (which we interpreted as within 10 minutes) so drugs cannot be prescribed straight away. At site 10, midwives monitor fetal heart rate in the second stage of labour only after each contraction, rather than every 5 minutes as recommended in national standards. At site 12, midwives do meconium observations 3-hourly rather than 2-hourly.

Staff also queried the internal logic of some standards; for example, one ND standard is to observe maternal temperature every 4 hours during the second stage of labour; but if this stage lasts 4 hours, this would not be a normal delivery.

Unpredictable resource requirements

The audit strand of this study was very time-consuming and, therefore, resource intensive. The main problem for those planning future studies is that resource requirements varied unpredictably between research sites. At some sites it was very difficult for local administrators or clinicians to identify which clinical records should be audited owing to deficiencies in admissions or discharge databases. In addition, ordering clinical records was more straightforward at some sites than at others. The administrators undertaking this work did so at the margins of their main role and frequently found it difficult to allocate time to seeking help to resolve difficulties, causing delay in the audit data collection. Although, as part of the process of negotiating recruitment of research sites, researchers tried to understand as much as possible about local contexts that would affect audit work, site-based informants rarely anticipated the difficulties auditors subsequently encountered. In most cases site-based informants were clinicians who frequently used and contributed to clinical records but, quite reasonably, lacked detailed knowledge of some administrative processes relating to clinical records. Advice from central audit departments was normally given by managers. It was difficult for researchers to interact with front-line auditors prior to data collection.

Clinical records at some sites were predominantly electronic and could be audited quickly; at other sites they were wholly paper based. For some conditions at some sites, the case notes that had to be accessed spanned two or three separate systems. At some sites, auditors had to retrieve and re-shelve their own notes from medical records departments; at others, they were retrieved for them. These factors greatly alter the time required for audits but are difficult to predict before an audit is under way, even though researchers always initiated pre-audit discussion of these matters with local clinicians and administrators. Trust-based auditors from different trusts needed strikingly different lengths of time to conduct audits of the same conditions. We cannot know the degree to which these differences reflected the efficiency and experience of the auditors, other workload demands faced by auditors, or site-specific challenges.

These unpredictable resource requirements contrast with more predictable resource differences relating to the nature of the audit standards. The standards we audited varied in their ease of extraction. For example, those relating to MSL required data extraction from both the mothers’ notes and the babies’ notes, which would be stored separately. It was necessary to audit the mothers’ notes first to identify the required baby notes. Another example is that certain clinical observations occur once and are easy to identify within case notes: this was the case with several ED audit standards. In contrast, the midwifery notes made during normal labour can form an extended narrative in which many clinical observations are embedded: data extraction requires extensive reading. These are largely predictable variations in resource requirements that can be built into study designs.

Factors influencing the indicators of organisational climate

Multilevel modelling of scores on the organisation scale (individual responses nested within research sites nested within hospitals, as described in Chapter 3, Multilevel modelling) showed that one site-level variable (the number of people included in the staff survey) and two individual-level variables (whether a manager and the number of years working in the trust) mediated organisation scores (Table 13). The model included 510 cases and estimated the underlying level of organisation scores as 2.672 (SE 0.119). The estimated parameters in Table 13 indicate that:

• Organisation scores were marginally higher in departments that distributed larger numbers of questionnaires, but only 0.0002 (estimated parameter × SD) of organisation scores higher for every additional 62 (1 SD) people included in the staff survey so this statistically significant result has no practical significance.

• Respondents who managed other staff returned organisation scores that were a little higher (0.033 higher) than the scores from those who did not manage others.

• Compared with respondents in the modal category of 6–10 years’ employment with their trust, respondents who had worked at the trust for < 6 years or > 15 years returned noticeably higher organisation scores [between 1.577 (1–2 years) and 0.552 (> 15 years) higher].

The multilevel model partitioned the remaining unexplained variation in organisation scores follows:

• 11% at hospital level (estimated variance 0.067, SE 0.039, χ² 2.959, 1 df, p = 0.085)

• 89% with respondents within sites (estimated variance 0.562, SE 0.035, χ² 251.0, 1 df, p < 0.0001).

There is significant residual variance in the organisation scores that cannot be accounted for by respondents’ demographic and role-related factors. This variation may relate to unmeasured individual characteristics but we regard it as mainly identifying site-level differences in perceptions of organisational climate.

A similar analysis was performed for the error wisdom scale. The multilevel model used 475 cases and estimated the underlying level of error wisdom scores at 2.805 (SE 0.106). The estimated parameters in Table 14 show that eight variables mediated error wisdom scores, as follows:

• Compared with staff in EDs, DU staff perceived higher levels of organisational error wisdom, 0.038 higher.

• As in previous analyses, there was a very small effect associated with the number of people included in the staff survey, which was too small to have practical significance, Error wisdom scores increased by 0.0003 for each additional 62 questionnaires distributed.

• At the individual level, age was a mediating factor: compared with the modal reference group aged 31–40 years, older respondents rated their trust’s error wisdom more highly: scores were 0.020 higher for the 41- to 50-year age group and 0.017 higher from those aged 51–65 years.

• People who had a mentor in the previous 12 months returned higher error wisdom scores, 0.018 higher.

• Managers rated their trust’s error wisdom more highly than non-managers, 0.017 higher.

• People who received updating or online training or who shadowed someone at some point during the previous year also rated their trust’s error wisdom more highly; scores were increased by 0.017, 0.017 and 0.016, respectively.

The multilevel model partitioned residual variation as follows:

• 10% at hospital level (estimated variance 0.038, SE 0.022, χ² 2.811, 1 df, p = 0.094)

• 90% at the level of respondents nested within sites (estimated variance 0.348, SE 0.023, χ² 233.6, p < 0.0001).

There remains significant residual variance in error wisdom scores, not accounted for by respondents’ demographic and role-related factors. Although the influence of unmeasured factors cannot be ruled out, this will be taken to signal significant variation in error wisdom scores between research sites; reflecting underlying variation in organisational culture.

Combining indicators of organisational climate

Responses to staff survey questions intended to elicit perceptions of organisational climate (described in Chapter 3, Developing the organisational and safety climate questionnaire) were combined as described in Chapter 3, Combining indicators of organisational climate. The mean and SD of the summary survey-based organisational climate scores were 0 and 0.752, respectively. In Table 15 the research sites have been arranged so that the highest scoring sites appear at the top: these are the sites where staff perceptions of organisational climate were most positive. It was noticeable that there were more delivery units than emergency departments in the positively scoring sites, where perceptions of organisational climate were above average, but analysis of variance showed that the difference between scores for delivery units and emergency departments was not significant (F = 2.115, df 1,14, p = 0.168). The scores in Table 15 also show that staff in different clinical services within the same hospital can have different perceptions of the organisational climate that they arguably share. The widest disparity occurred at hospital 4 and the smallest difference at hospital 1.

Comparing indicators of safety climate

Responses to the two British scales, overload and line management, exhibited a low level of positive correlation [Pearson’s r = 0.285 (n = 497), p < 0.0001], but the score on one scale explains only 8% of the variation in scores on the other scale (r² = 0.081). In contrast, the two scales developed in the USA were strongly correlated [r = 0.757 (n = 485), p < 0.0005]: evaluations of teamwork climate and safety climate move in step with one another and the score on one explains 57% of the variation in the other (r² = 0.573). Furthermore, Tables 18 and 19 show that teamwork climate and safety climate were scored at similar levels (mean 3.99, SE 0.038, and mean 3.79, SE 0.036, respectively).

Correlations between scores on the British overload scale and the American scales were low (0.320 and 0.325, p < 0.0001; Table 20). To improve readability, entries in the lower triangle of Table 20 have been removed because they duplicate entries in the upper triangle. Hence, low levels of reported overload are to some degree associated with high scores for teamwork and safety climates, but overload scores only account for 10% of the variation in teamwork climate scores (r² = 0.102) and for 11% of the variation in safety climate scores (r² = 0.106). There were stronger correlations between the British line management scale and the American scales (0.570 and 0.600, p < 0.0005; see Table 20). Positive evaluations of supportive line management are associated with higher scores for teamwork and safety climates. Line management scores account for 32% of the variation in teamwork climate scores (r² = 0.325) and 36% of the variation in safety climate scores (r² = 0.036).

The reverse-coded single item ‘Relationships at work are strained’ was moderately correlated with all four scales, in the range 0.428–0.502 (p < 0.0005; see Table 20): perceiving strained relationships is somewhat linked to perceiving overload and lower evaluations of line management, teamwork and safety climates, accounting for between 18% and 25% of the variation in the scores on these scales (the range for r² was 0.183–0.252).

Factors influencing the indicators of safety climate

Multilevel modelling of the scales intended to capture aspects of safety climate showed that four respondent-level factors (professional and ethnic groups, years worked for trust and having a mentor at some time in the previous year) mediated scores on the overload scale (Table 21). The model used 472 cases and the estimated underlying level of overload scores at 2.812 (SE 0.172). The estimated parameters in Table 21 show that:

• Compared with the modal reference group ED nurse or DU midwife, respondents whose roles supported care rather than directly provided care (e.g. administrative and clerical staff, non-clinical managers, porters and housekeepers) reported lower levels of overload. The estimated difference in scores was 0.112.

• Compared with respondents in the reference group of 6–10 years working for the trust, which was the modal category and contained the median duration of employment, respondents who had worked for their trust for < 3 years reported lower levels of overload. The estimated difference in scores was 0.068 for those employed for 1–2 years and 0.070 for those employed for < 1 year.

• Compared with the reference category (and modal response) of White ethnicity (see questionnaire in Appendix 3 for finer-grained composition of ethnic grouping used by the NHS) respondents identifying with any of the categories within the Asian or Asian British ethnic group reported lower levels of overload. The estimated difference in scores was 0.065.

• Respondents who had a mentor reported lower levels of overload than those who did not. The estimated difference in scores was 0.061.

The multilevel model, which included these fixed factors, partitioned variance as follows:

• 3% at hospital level (estimated variance 0.020, SE 0.017, χ² 1.394, 1 df, p = 0.238)

• 97% with respondents within research sites (estimated variance 0.734, SE 0.048, χ² 232.0, 1 df, p < 0.0001).

Although there is no discernible hospital-level variation in perceptions of overload, there is significant variation in responses to the overload scale from respondents nested within research sites, which remains after including the mediating effects of professional and ethnic groups, duration of employment, managerial responsibility and engagement with CPD. Perceptions of overload varied between the clinical departments participating in this study.

Multilevel modelling of line management revealed that one site-level variable (response rate) and six respondent-level variables (professional group, being a manager, gender, receiving on-the-job training, having a mentor and updating knowledge or skills scores) mediated scores on this scale. The best-fitting model included 477 cases and the underlying level of line management scores was estimated at 2.496 (SE 0.191). The parameter estimates for mediating variables, listed in Table 22, show that:

• Research sites with above average response rates returned lower line management scores (0.289 lower).

• Compared with the modal reference group of DU midwives and ED nurses, doctors returned higher line management scores (0.081 higher).

• Respondents who had been supported to update skills or knowledge during the past year also returned higher line management scores (0.062 higher).

• Managers and women reported more favourable views of their direct managers, with line management scores 0.054 and 0.050 higher than non-managers and men, respectively.

• People who had received on-the-job training or had a mentor at some point during the past year were also more positive about their line managers, with scores, respectively, 0.048 and 0.046 higher.

The multilevel model partitioned variance as follows:

• 13% at hospital level (estimated parameter 0.108, SE 0.061, χ² = 3.116, 1 df, p = 0.078)

• 87% with respondents within sites (estimated parameter 0.725, SE 0.047, χ² = 253.5, 1 df, p < 0.0001).

Most of the variation that remains after including the mediating effects of response rates, professional groups, managerial responsibility, gender and engagement with CPD lies at the level of respondents nested within research sites. Respondents from different research sites perceived the quality of their line management differently.

Multilevel modelling of the teamwork climate scale developed by Sexton and colleagues25 showed that one site-level variable (the number of people included in the staff survey) and seven respondent-level variables (professional group, gender, years employment with trust, three types of CPD and being a manager) mediated teamwork climate scores (Table 23). The model used 455 cases and the estimate of the underlying level of teamwork scores was 3.098 (SE 0.176). The parameter estimates show that:

• Compared with the reference group of ED nurses and DU midwives, doctors’ teamwork climate scores were 0.077 higher.

• Women and managers had higher teamwork climate scores, 0.056 and 0.036 higher than men and non-managers, respectively.

• Longer than average employment with the trust was also associated with higher teamwork climate scores. Compared with the modal reference group who had worked for their trust for between 6 and 10 years, respondents who had been employed for 11–15 years returned scores 0.050 higher, and for those with more than 15 years’ service the increase was 0.065.

• Updating knowledge or skills, having a mentor and doing online training each increased teamwork climate scores by between 0.038 and 0.039.

• The site-level variable of the number of people identified to receive the staff survey had a very small positive effect, adding 0.00005 to scores for every additional 62 (1 SD) people included. This small increase has no practical significance.

The best-fitting model partitioned residual variance as follows:

• 12% at hospital level (estimated parameter 0.072, SE 0.041, χ² = 2.994, p = 0.084)

• 88% with respondents nested within research sites (estimated parameter 0.525, SE 0.035, χ² = 223.5, p < 0.0001).

Significant variation in the Sexton teamwork climate scores remains with respondents nested within the research sites. Although effects due to unmeasured individual-level variables cannot be ruled out, the result will be interpreted as identifying site-level variation in teamwork climate among the clinical departments that participated in this study.

Finally, we examined scores for Sexton and colleagues’ safety climate scale. 25 Table 24 shows that seven respondent-level variables mediated these scores (being a manager, updating skills or knowledge, age, professional group, receiving on-the-job training and having a mentor). The best-fitting model used 462 cases and the underlying estimate for Sexton safety climate was 2.879 (SD 0.153). The parameters in Table 24 show that:

• Managers evaluated safety climate more positively than staff who did not manage others, scoring 0.049 higher.

• Respondents who had been supported to update their knowledge or skills during the past 12 months returned higher safety climate scores (0.048 higher).

• Staff who were older than the modal reference category of 31–40 years returned higher safety climate scores (0.027 higher for those aged 41–50 years and 0.048 higher for those aged 51–65 years).

• Women evaluated safety climate more highly than men, scoring 0.040 higher.

• Staff who had received on-the-job training or had a mentor at some point during the past year returned safety climate scores that were, respectively, 0.033 and 0.026 higher.

The model that contained the fixed factors shown in Table 24 partitioned residual variation as follows:

• 6% at hospital level (estimated parameter 0.033, SE 0.022, χ² = 2.300, 1 df, p = 0.129)

• 94% with respondents nested within research sites (estimated parameter 0.517, SE 0.034, χ² = 227.1, p < 0.0001).

After adjusting for respondent-level mediating factors, the scale developed by Sexton and colleagues25 shows significant site-level residual variation in safety climate scores.

Combining indicators of safety climate

Summary survey-based safety climate scores were calculated as described in Chapter 3, Combining indicators of safety climate. The results are displayed in Table 25, which has been arranged so that the highest-scoring sites appear at the top: these are the sites where staff perceptions of the elements forming the summary safety climate score were most positive. The mean of the site-specific survey-based safety climate scores is zero (SD 0.835). Echoing the summary survey-based organisational climate scores (see Table 15), site 6 is at the top of the table, reflecting positive staff perceptions, whereas site 9 lies at the bottom.

Acute coronary syndrome

The audit standards were:

1. Electrocardiography performed immediately.

2. Aspirin should be given immediately if not already given by ambulance service.

3. If patient in pain, times from arrival to the administration of pain relief:

   1. – in cases of severe pain, 50% in 20 minutes, 75% in 30 minutes, 98% in 60 minutes

   2. – in cases of moderate pain, 75% in 30 minutes, 98% in 60 minutes.

Table 29 shows the percentage of audited cases for which the clinical records contained evidence of meeting the standards in these audit criteria (ACS1 and ACS2) and degree to which the targets in ACS3 were met, expressed as a percentage of fully meeting the target. Table 29 also shows the number of audited cases to which this standard applied. For ACS1, ‘immediately’ was interpreted as within 10 minutes of arrival. Compliance with the ACS1 standard ranged from 14% to 48%: most patients do not have an ECG in the first few minutes after arrival. The ACS2 marker proved problematic. It did not apply to a large number of audited cases because aspirin had already been given by the ambulance service or the patient was taking aspirin daily. Audited notes were also checked for recorded allergies and any other noted contraindications. Irrespective of this, there was very low compliance with this standard for the cases to which it applied, ranging from zero to 27%. ACS3 required an assessment of pain for all patients and, when moderate or severe pain was detected, early administration of pain relief. Compliance with the ACS3 standard ranged from 7% to 62%: many patients experienced delays before receiving pain relief.

Acute severe asthma

The audit standards were:

1. Oxygen saturation should be measured and recorded on arrival in 98% of cases.

2. Salbutamol or terbutaline should be given within 10 minutes of arrival.

3. Intravenous hydrocortisone or oral prednisolone should be given within 30 minutes of arrival in 90% of cases.

4. Measurement of oxygen saturation should be repeated within 60 minutes in 75% of cases.

There was a change in the guidelines for ASA after data collection at site 1. 182,207 In the earlier guidance, ipratropium was recommended both for moderate asthma, and for acute severe and life-threatening asthma; in the later guidance, for life-threatening asthma only. The new guideline groups these two types of asthma together, but our inclusion criteria were unable to distinguish between acute severe and life-threatening, and therefore had to omit this from our markers. In applying the new standards to sites 3–15, we therefore overlook the fact that life-threatening cases require higher standards than those we are applying: we treat them all as severe.

Table 30 shows the audit results for ASA. In ASA1, ‘on arrival’ was interpreted as within 10 minutes and compliance varied between 40% and 84%; this baseline observation was missing from many patient records. Compliance with audit standard ASA2 was much lower (range 6%–36%), with six EDs scoring no more than 20%. The degree to which the 90% target in audit standard ASA3 was met varied between 14% and 43%. There was wide variation in the degree to which research sites met the target in audit standard ASA4: two EDs scored < 20%, whereas the others scored in the range 43–69%.

Fractured neck of femur

The audit standards were:

1. Pain score recorded on arrival.

2. Time from arrival to receive analgesia:

   1. if severe pain, 50% in 20 minutes, 75% in 30 minutes, 98% in 60 minutes

   2. if moderate, 75% in 30 minutes, 98% in 60 minutes.

3. Radiography performed within 60 minutes of arrival in 75% of cases.

4. If in pain, evidence of re-evaluation of pain (90% of those with severe pain re-evaluated within 30 minutes; 75% of those in moderate pain within 60 minutes).

5. Admitted within 4 hours of arrival.

In audit standard FNoF1, ‘on arrival’ was interpreted as within 10 minutes, and Table 31 shows that, although five EDs scored in the range 80–92%, levels of compliance fell as low as 11% elsewhere. The degree to which the targets for administering analgesia without unnecessary delays (FNoF2) were met mainly varied between 10% and 51%, with research site 9 scoring more highly at 76%. Compliance with the target for timely radiography (FNoF3) varied widely, between 11% and 97%, with half of the EDs scoring over 72%. Research site 9 had the highest level of compliance with FNoF3 (97%) but the lowest level of compliance with FNoF1 (11%). Levels of recorded compliance with FNoF4, the timely re-evaluation of pain, were extremely low: zero for five EDs and elsewhere no more than 4% in cases to which this standard applied. In half of the EDs, admission within 4 hours (FNoF5) was achieved in over 85% of cases. Elsewhere the level of compliance dropped as low as 13%. It was noticeable that research site 1 had a high level of compliance with FNoF1 and an exceptionally low level of compliance with FNoF5, whereas the reverse was true for research site 13. These within-condition variations are smoothed out by the calculation of condition-specific means and further aggregation in Summarising the site-specific audit results. This will be discussed in Chapter 9.

Normal delivery

The audit standards were:

1. Initial assessment should include temperature, pulse, blood pressure, urinalysis, length, strength and frequency of contractions, fundal height, lie, presentation, position and station, show, liquor, blood, pain and fetal heart rate.

2. Assessment during first stage should include fetal heart rate every 15 minutes, frequency of contractions every 30 minutes, pulse hourly, temperature and blood pressure 4-hourly, vaginal examination offered 4-hourly, frequency of emptying bladder.

3. Assessment during second stage should include fetal heart rate every 5 minutes, frequency of contractions every 30 minutes, pulse and blood pressure hourly, vaginal examination offered hourly, frequency of emptying bladder, woman’s emotional/psychological needs.

Delivery unit midwives rarely recorded all the observations in the multifaceted audit standards for initial assessment and assessments during the first and second stages of labour. We will discuss this in Chapter 9. Table 32 shows that compliance with audit standard ND1 was zero at all sites. The most frequently missing observations were blood and show. In half of the DUs, compliance with audit standard ND2 was between 21% and 26%, but elsewhere it was < 12%. Compliance with ND3 mostly varied between 17% and 33%, with research site 16 scoring only 2%. The applicability of some markers for standards ND2 and ND3 depended on the length of stages 1 and 2 of labour, respectively; where standards were inapplicable, cases were excluded from the denominator.

Emergency caesarean section

The audit standards were:

1. Evidence of consultant obstetrician involvement in the decision to perform an ECS should be documented.

2. Women undergoing an ECS should be offered prophylactic antibiotics at the time of ECS.

3. All women undergoing ECS must receive thromboprophylaxis for VTE.

4. Women undergoing ECS should be offered regional anaesthesia (spinal or epidural).

Table 33 shows that in six DUs audit standard ECS1 was met in the majority of cases (range 55–84%); elsewhere levels dropped as low as 26%. Half of the DUs had high levels of compliance with audit standard ECS2 (81–98%) and the other half had moderate levels (48–55%). Compliance with audit standard ECS3 was generally high, between 92% and 100% at seven research sites, but was only 45% at research site 4. Discounting the very small number of cases for which regional anaesthesia was clinically inappropriate, compliance with audit standard ECS4 was high (range 92–100%).

Meconium-stained liquor (grade 2 or 3)

The audit standards were:

1. Continuous electronic fetal monitoring (EFM) should be advised.

2. Health-care professionals trained in advanced neonatal life support should be available for the birth.

3. Baby assessment should include at 1 and 2 hours and then 2-hourly for 12 hours:

   1. – general well-being

   2. – chest movements and nasal flare

   3. – skin colour including perfusion

   4. – feeding

   5. – muscle tone

   6. – temperature

   7. – heart rate and

   8. – respiratory rate.

Table 34 shows that continuous electronic fetal monitoring (MSL1) was arranged in nearly all cases (range 81–100%); in many cases, this was already in place when meconium was detected. Compliance with audit standard MSL2 varied much more widely: between 69% and 98% in five DUs and dropping as low as 6% elsewhere. Similar to the results for the multifaceted ND audit standards, very few babies received the multifaceted observations required by MSL3. The most commonly missing observations were feeding and muscle tone.

H1a: organisational climate and culture assessments

Hypothesis 1a (Chapter 1, Hypotheses) was ‘There will be a strong correlation and good agreement between questionnaire-based and holistic evaluations of organisational culture’. This was investigated using the summary survey-based organisational climate z-scores (Table 15 and reproduced in Table 36) and the summary observation-based organisational culture scores from Table 28 after they had been scaled to the standard normal distribution (see Table 36).

Visual comparison of the organisational climate and culture scores can be made by viewing the scatterplot in Figure 9. As noted in Chapter 6, Scoring, there is some concern about the number of ties in the observation-based scores. These show up in Figure 9 as three vertical stacks of data points: sites for which the observation-based measurement resulted in the same total score for organisation and work environment factors. Visual inspection of Table 36 and Figure 9 also highlights that seven research sites had an above average (positive) score on one measurement and a below average (negative) score on the other measurement, while nine sites had two positive or two negative scores, signalling a likely lack of correlation between these measures. As expected from the visual inspections, the correlation between the organisational climate and culture scores in Table 36 was low (r = 0.252) and not statistically significant (p = 0.346). In addition, the correlations for the prevalidated scales contributing to the summary organisational climate score (Chapter 3, Combining indicators of organisational culture) both had low correlations with the observation-based organisational culture score (organisation, r = 0.236; error wisdom, r = 0.108).

FIGURE 9.

Scatterplot of standardised summary organisational climate and culture scores. Data points have been annotated with the research site numbers.

The SD of differences plotted in Figure 10 is 1.089. Several differences cluster close to zero, but there is not a clear pattern to the distribution of differences. There is one point outside the ‘limits of agreement’ (mean difference ± 2SD). This relates to research site 10, where researchers assessed organisation and work factors as deficient but staff perceived the organisational climate more positively. The two scales contributing to the survey-based organisational climate score were also compared with observation-based organisation culture scores using Bland–Altman plots and neither offered notably better agreement.

FIGURE 10.

Bland–Altman plot for survey-based organisational climate and observation-based organisational culture scores.

Those aspects of organisational culture that could be observed during this study (the adequacy of staffing and premises, availability of sufficient functioning equipment, the availability and quality of support from administrative staff, senior managers and other supporting teams) did not correlate with staff perceptions of organisational climate, where the survey instruments focused on perceptions of senior management, organisational communication, patient care, whistle-blowing and error-reporting procedures and organisational error wisdom. These matters were touched upon in the brief key informant interviews that contextualised observations but they could not be observed directly. The level of agreement between the survey-based and observation-based measurements was reasonably good; only one point lay outside the limits of agreement (mean difference ± 2SD). Some clustering of points is visible, created by tied scores (lack of discrimination) in the observation-based measure.

H1b: Safety climate and culture assessments

Hypothesis 1b proposed that ‘There will be a strong correlation and good agreement between questionnaire-based and holistic evaluations of safety culture’. This was investigated using the summary survey-based safety climate z-scores (Table 25 and reproduced in Table 37) and both the observation-based teamwork and holistic safety culture scores (Table 28 and, after scaling to standard normal distribution, z-scores in Table 37). Visual comparisons can be made by inspecting Figures 11 and 12. As in H1a: organisational climate and culture assessments, visual inspections of the scatterplots and Table 37 signalled that correlations would be low.

FIGURE 11.

Scatterplot of standardised survey-based safety climate and observation-based holistic safety culture scores. Data points are annotated with research site numbers.

FIGURE 12.

Scatterplot of standardised survey-based safety climate and observation-based teamwork culture scores.

The correlation between the summary survey-based safety climate and observation-based holistic safety culture z-scores was low (r = 0.345) and not statistically significant (p = 0.191, n = 16; see Figure 11). Similarly, the correlation between the summary survey-based safety climate and observation-based teamwork culture z-scores was low (r = 0.316) and not statistically significant (p = 0.234, n = 16; see Figure 12). Moving to agreement, the SD of differences in Figure 13 was 1.059. There was no discernible pattern, and only one point lay close to the upper limit of agreement (research site 8, where researchers evaluated the safety culture deficient but staff returned average safety climate scores). Thus, there was good agreement between the contrasting measurements of safety climate and holistic safety culture. Ordinal evaluations of those aspects of safety culture that could be observed during this study (safety-related organisation and work environment factors; aspects of team members’ interactions) agreed but did not correlate with staff perceptions of safety climate, where the survey questions focused on perceptions of overload, line management, teamwork, interactions between team members, the safety of care and management, organisational communication, patient safety concerns and responses to errors. There was some overlap between the components of these measurements. Similarly, the SD of differences in Figure 14 was 1.082, there was no discernible pattern, and again the point relating to research site 8 was the only one close to a limit of agreement (because observers judged safety-related teamwork factors to be poor but staff returned average safety climate scores). Overall, agreement was good.

FIGURE 13.

Bland–Altman plot for survey-based safety climate and observation-based holistic safety climate.

FIGURE 14.

Bland–Altman plot for survey-based safety climate and observation-based teamwork culture.

The prevalidated scales forming parts of the summary survey-based safety climate measure [overload, line management, teamwork climate (Sexton et al. 25) and safety climate (Sexton et al. 25)] were also compared with observation-based assessments of culture but none of these offered improved agreement.

H3a(i): comparing survey-based organisational climate with audit assessments of the quality of care

The first part of hypothesis 3a (Chapter 1, Hypotheses) proposed that ‘There will be a moderately strong correlation and reasonably good agreement between criterion-based measurements of the quality of care and questionnaire-based evaluations of organisational culture’. This was investigated using the summary survey-based organisational climate z-scores (see Table 15) and the summary audit z-scores (see Table 35). The correlation between these measures was negligible (r = –0.096), which is evident from visual inspection of Figure 15. Moving to agreement, the SD of differences in Figure 16 is 1.058; there is no discernible pattern but two points lie on the limits of agreement. There is reasonably good agreement between these measures.

FIGURE 15.

Scatterplot of standardised summary survey-based organisational climate and audit scores. Data points are annotated with research site numbers.

FIGURE 16.

Bland–Altman plot for survey-based organisational climate and criterion-based audit.

The prevalidated scales contributing to the summary survey-based organisational climate scores ‘organisation’ and ‘error wisdom’ were also compared separately with the criterion-based audit scores, neither displayed better agreement.

H3a(ii): comparing observation-based organisational culture with audit assessments of the quality of care

According to the second part of hypothesis 3a, ‘There will be a moderately strong correlation and reasonably good agreement between criterion-based measurements of the quality of care and holistic evaluations of organisational culture’. This was investigated using the observation-based organisational culture (see Table 28) and summary audit z-scores (see Table 35). There was a moderate negative correlation between these measures (r = –0.481, p = 0.059, n = 16), although visual inspection of Figure 17 reveals that there can be considerable variation in audit scores among research sites with the same observation-based organisational culture scores and that the results from research sites 6 and 10 may be having a disproportionate effect on the correlation coefficient. Removing research sites 6 and 10 from the analysis reduces the correlation to r = –0.347 (p = 0.225, n = 14). Moving to agreement, the SD of differences in Figure 18 is 1.451, tied observation scores are linked to a pattern of three diagonal lines within the points within the limits of agreement (mean difference ± 2SD), and the points relating to research sites 6 and 10 are once again anomalous, lying, respectively, above and below the limits of agreement. At research site 6 researchers perceived a positive organisational culture but audit scores were below average. Conversely, researchers awarded site 10 the lowest score for organisational climate but audit scores were above average. Agreement between these measures is poor.

FIGURE 17.

Scatterplot of standardised summary observation-based organisational culture and audit scores. Data points are annotated with research site numbers.

FIGURE 18.

Bland–Altman plot for observation-based organisational culture and criterion-based audit.

H3b(i): comparing survey-based safety climate with audit assessments of the quality of care

According to the first part of hypothesis 3b, ‘There will be strong correlations and good agreement between criterion-based measurements of the quality of care and questionnaire-based evaluations of safety culture’. This was investigated using the summary survey-based safety climate (see Table 25) and summary audit z-scores (see Table 35), which are displayed in Figure 19. The correlation between these sets of scores was very low: 0.150. Moving to agreement, the SD of differences in Figure 20 is 0.992, there is no discernible pattern to the points, and one point lies outside the limits of agreements and one point lies on the upper limit of agreement. Agreement between these measures is reasonably good.

FIGURE 19.

Scatterplot of standardised summary survey-based safety climate and audit scores. Data points are annotated with research site numbers.

FIGURE 20.

Bland–Altman plot for survey-based safety climate and criterion-based audit.

The four prevalidated scales (overload and line management,175 and teamwork and safety climates25) were also investigated and none provided better agreement with audit scores. Correlations also remained low, maximum 0.209.

H3b(ii): comparing observation-based safety culture with audit assessments of the quality of care

The second part of hypothesis 3b proposed that ‘There will be strong correlations and good agreement between criterion-based measurements of the quality of care and holistic evaluations of safety culture. This was investigated using the observation-based teamwork and holistic safety culture z-scores (see Table 37) and the summary audit z-scores (see Table 35). The comparison between observation-based holistic safety climate scores and audit scores found low negative correlation (r = –0.201; Figure 21), and research sites 7 and 8, at the same hospital, appear to be outliers: researchers awarded very low scores for safety culture but audit results were around average. Moving to agreement, the SD of differences in Figure 22 is 1.314 and the points relating to research sites 7 and 8 lie towards the bottom-left of the graph. There is no discernible pattern to the remaining differences and all points lie within the limits of agreement. Overall, the agreement between these two measures is reasonably good.

FIGURE 21.

Scatterplot of standardised summary observation-based holistic safety culture and audit scores. Data points are annotated with research site numbers.

FIGURE 22.

Bland–Altman plot for observation-based holistic safety culture and criterion-based audit.

The correlation between observation-based teamwork culture scores and summary audit scores was negligible (r = –0.062; Figure 23), and research sites 7 and 8 remain outliers, with the researchers perceiving poor teamwork culture, whereas audit results were around average. Moving to agreement, the SD of differences in Figure 24 is 1.241. The point relating to research site 7 now lies very close to the lower limit of agreement although all points are between the lines. There is no pattern to give rise to concern. The agreement between these measurements is reasonably good.

FIGURE 23.

Scatterplot of standardised summary observation-based teamwork culture and audit scores. Data points are annotated with research site numbers.

FIGURE 24.

Bland–Altman plot for observation-based teamwork culture and criterion-based audit.

Aims

The study’s aims were to compare:

• questionnaire and holistic (observation-based) assessments of organisational and safety climate/culture and

• these assessments of organisational and safety climate/culture with criterion-based assessment of the quality of care.

These comparisons were made and the empirical results were reported in Chapter 8. Whether such comparisons are legitimate and useful are matters of epistemology and practicalities. The study’s three strands are epistemologically and empirically different:

1. self-reported perceptions, recorded in a staff survey

2. non-participant observation made mainly by non-clinical health services researchers, supported by nurses, midwives and a service-user and

3. criterion-based, retrospective case note audit by non-clinical auditors.

Strand A staff perceptions may be viewed as qualitative data, which the survey participants quantified as they responded to the Likert scales in the questionnaire. Strand B observations were recorded in semistructured field notes and contained qualitative evaluations of behaviours, workplace practices and facets of the physical environment that were pertinent to safety (see data collection prompt list, Appendix 5). Researchers subsequently quantified these qualitative evaluations using the approach described in Chapter 4, Quantifying semistructured observations (which will be discussed in Tensions and challenges in quantifying observational data). Case notes contain a mixture of qualitative and quantitative records made by health-care staff describing observations, requests and decisions made during the care process, and actions taken. Criterion-based audit (strand C) records the presence, partial presence or absence of selected indicators of the quality and safety of care, thus synthesising qualitative and quantitative notes into quantitative data.

The three strands of this study powerfully illustrate the process of reducing qualitative evaluations to quantitative representations to facilitate quantitative analyses and comparisons, but it is acknowledged that the reductionism in this process discards or disguises the nuances that aid understanding of the numerical results. For this reason, in earlier work,132 we have treated qualitative and quantitative observations of workplace practices as complementary to support a better-elaborated understanding than either approach delivered alone. In contrast, the purpose of this study was to ‘triangulate’ (commissioning brief, see Appendix 1) in the sense of examining the degree to which different data sets provide the same understanding of workplace practices.

In multimethod research213 (sometimes termed mixed methods214 research) integration of different types of data is most commonly thought of in terms of complementarity, and Brannen215 (p. 13) notes that ‘differences between data sets are likely to be as illuminating as their points of similarity’. Because this study was commissioned to test similarities, it tends to position differences as problems. However, we continue to view differences as illuminating and worthy of further research. The observation field notes from strand B provide illumination of the distribution of points in Figures 9–24 which cannot be included in this report but which we hope to publish elsewhere.

Objectives

1. Objective 1: To work with staff in the participating trusts and user stakeholders such that their organisational and professional knowledge is respected, the study is understood and supported within participating departments, and prompt feedback to participant departments allows local development in advance of the study reporting.

Examples of fulfilling this objective include asking clinical leads at each research site to define the categories of staff to be included in the staff survey, with minimal exclusions imposed by the research team (see Chapter 3, Sampling, delivery and maximising returns). Early feedback was offered to clinical teams at all (16) research sites and taken up by 10. Feedback included additional contextual data for clinical teams to support local quality improvement, but not included in this report as they were not necessary for the methodological comparisons which form the report’s focus. According to local preferences, feedback was sometimes provided to a small group of senior staff and sometimes presented during a naturally occurring meeting of the wider clinical team.

1. Objective 2: To use questionnaires to obtain quantitative assessments of the organisational and safety climate at each site.

Questions from the NHS annual staff survey175 and teamwork and safety climate scales25 formed the staff survey questionnaire used in this study (see Chapter 3, Developing the organisational and safety climate questionnaire). Prevalidated scales were first examined separately, then weighted averages were calculated for the responses to questions selected to elicit aspects of organisational climate and safety climate, as described in Chapter 3, Combining indicators of organisational climate and Combining indicators of safety climate. This process created summary survey-based assessments of organisational climate and safety climate. The questionnaire was shortened after slightly disappointing response rates at research sites 1 and 2, and consequently fewer questions contributed to the summary survey-based assessment of organisational climate than to the summary survey-based assessment of safety climate. The former comprised perceptions of senior management, organisational communication and error wisdom. The latter comprised perceptions of overload, line management, team work and safe care. The results of the staff survey were reported in Chapter 5.

The staff survey yielded variable response rates, although less variable than elsewhere. 216 The overall response rate was 27.6% and the range of site-specific response rates was 9–47% [interquartile range (IQR) 23–36%]. Extra measures (beyond salience, basic design, piloting, provision of addressed response envelopes, incentive in the form of a prize draw and a reminder after 2 weeks) were taken to improve response rates, including shortening the questionnaire and offering an online version of the survey in parallel with the postal version. Neither of these measures had a discernible effect on response rates, which continued to vary unpredictably. However, 531 completed questionnaires were received and analysed for this report, which includes comparisons to gauge whether the study respondents were broadly representative of the NHS workforce as a whole (see Chapter 5, Demographics and role-related characteristics, and objective 8 below).

Although the overall response rate was similar to other UK patient safety studies,100,178 at each research site the study response rate was lower than trust-wide response rates for the longer NHS annual staff survey (see Chapter 4, Samples and response rates). However, the small, trust-wide samples of staff randomly selected to receive the NHS national staff survey are more strongly encouraged to complete and return it than would be regarded as ethical in research studies. Encouragements include multiple reminders and appeals from all levels of management, often including the trust chief executive. Similarly, some patient safety studies report considerably higher response rates than those obtained in this study. 96,106,217 In these studies, questionnaire distribution was managed by hospital staff who took the role of champions for the studies and were active in increasing response rates. This raises concerns about coercion. For example, Holden and colleagues218 obtained response rates of 59.6% and 54.2% in a two-hospital study, but potential participants were handed an individually addressed survey pack during in-service training, staff meetings or shift handovers. The pack contained a small cash incentive and was followed up with three reminders over a 3- to 4-week period.

Site-specific response rates were included as possible explanatory variables in multilevel modelling of factors that may mediate scores on the prevalidated scales included in this study’s survey. Response rate was not a significant factor. This echoes findings of another recent study,216 which experienced even more variable response rates to the (hand-delivered) SAQ25 from 63 surgical departments (range 5–100%, IQR 25–64%). Watts and colleagues216 included response rates in a regression analysis and found that response rates were not significantly correlated with scores on any of the six SAQ domains (two of which, teamwork climate and safety climate, formed part of this study’s summary survey-based safety climate measure).

In our survey, responses to prevalidated scales drawn from the annual NHS staff survey were normally distributed, but there was some evidence of a degree of negative skew (a high proportion of positive responses) within responses to the teamwork and safety climate scales. 25 These teamwork and safety climate scales, developed in the USA and benchmarked more widely,96 address matters such as the coordination of different professional teams’ contributions to care, speaking up, questioning, resolving conflict, responding to errors and personal feelings about the safety of care. In recent years these matters have become basic tenets of quality improvement initiatives and a strong focus within initial professional education and CPD for health-care staff. 219–221 This could have sensitised respondents to ‘correct’ or socially desirable answers. The slight skew towards positive responses for the Sexton teamwork and safety climate scales could be limited evidence of improvements in these aspects of safety culture or limited evidence of social desirability bias. 26 Further research will be needed to disentangle these matters.

1. Objective 3: To generate quantified holistic evaluations of organisational and safety culture for each site using observation.

No established method of data collection existed for objective 3, so a suitable framework and scoring system was developed during the study. This was achieved using semistructured observations and strategic immersion in the ebb and flow of activity within the purposively selected clinical departments. Two experienced research fellows separately conducted non-participant observations at each research site during six half-day visits. In recognition of fluctuating workloads and staffing, the visits included 6 days (Monday–Saturday) and time sampling in the period 6 am to midnight (see Chapter 3, Data collection, for further details of the non-participant observation). At eight research sites additional 1-day visits were made by service provider researchers. This provided a clinical gaze to complement the non-clinical researchers’ understandings of what was being observed. This increased researchers’ confidence that they were not overlooking or misunderstanding important features of the workplace activity. At sites where service provider researchers were unable to attend, research fellows discussed their observations in detail with service provider researchers. Selected issues were also discussed with the project’s senior clinical advisors.

At three research sites, a service user observer made observations during one half-day visit. Her observations were valued by the research fellows to check that they had not ‘gone native’, in the anthropological sense, through gradually developing too close an identification with the clinical perspective. The service user observer also proved exceptionally good at focusing on safety-related non-technical skills12 exhibited by staff during their work. A service user observer could only be present in three research sites, owing to difficulties in recruitment of interested individuals and subsequent difficulties in securing research governance approvals for their participation (see Chapter 4, Recruiting service user and service provider coresearchers). Plans to streamline research governance processes and speed recruitment to studies may help future studies. 196,222

The value of ethnography in patient safety research is recognised:210 our use of strategic immersion is less established. The strengths of the semistructured observations and strategic immersion used in this study include economy and unobtrusiveness. Compared with much more prolonged ethnography, strategic immersion allows less time (in the form of fewer visits) for staff to become accustomed to the presence of observers such that the latter become ‘invisible’ so, in theory, reactivity to researchers may be present in a higher proportion of the observation data set. However, reactivity to researchers subsides very quickly in busy clinical environments: well within the half-day observation periods used in this study. Making more or fewer visits would not be expected to change the extent of researcher reactivity because staffing for each shift would be different. Limitations of less intensive observation included fewer opportunities to have informal conversations with members of the clinical team, which may have added more texture and greater discrimination to the observations made.

Sequential observation at the eight participating hospitals allowed researchers to form, test and consolidate observation-based assessments of cultures within each research site with the minimum risk of contamination from observations at other sites.

Once observations were complete at all research sites, two research fellows separately read all observation materials and notes from associated orientation interviews and feedback meetings. This included personal data collection and data collection by the other research fellow, service-user and service provider coresearchers. Each research fellow ascribed scores for every research site, using a four-point scale across eight categories. The categories and scoring system were developed during this study (see Chapter 3, Analysis, and Appendix 6). After independent scoring, the research fellows met to compare scores, discuss discrepancies and agree final scores. Fifty-eight per cent of the research fellows’ independent scores were in agreement. All scores were agreed after a short discussion. Five-point and six-point scoring systems were also trialled, but this study’s research fellows found it impossible to make confident and consistent assessments with any greater level of discrimination than the four-point scale reported here. As a result of this and the process of adding scores across eight categories (three relating to organisation and work environment factors and five relating to team factors), the scores for several research sites tied. This was a particular problem for the organisation and work environment factors. In future research it may be possible to obtain greater discrimination by narrowing the scope of what is observed. That would represent a trade-off between greater discrimination among observed clinical departments but narrower understanding of facets of safety culture.

Researchers were cautious about ascribing scores for research sites 1 and 2, where observations were made by two project research fellows who then left the study. One of the former research fellows joined discussions, and minor doubts about scoring were soon resolved. For future research this highlights that, ideally, scoring should be conducted by those who observe. However, detailed input from the former research fellow was only essential for 2 of the 16 scores required for sites 1 and 2, demonstrating that experienced researchers, using a sufficiently structured scoring system, can score from observation records made by others. Where staff changes are inevitable, a variety of mechanisms may help increase confidence in scoring, for example ongoing contact between outgoing and incoming research staff, as in this study; review and annotation of observation records by departing staff at the time they leave the study (done in this study); a formal handover and discussion of observation records; and more structured observation records (something which would have been difficult in this particular study as this strand was mainly exploratory to investigate whether semistructured observation could yield trustworthy quantifiable evaluations of visible aspects of safety culture).

How much is sufficient observation to inform a secure assessment of organisational and safety culture? This is equivalent to asking the broader research question regarding data saturation in qualitative research. In qualitative studies diminishing returns from continued data collection prompt withdrawal from research environments, but there can never be a guarantee that the next observation visit would not have revealed something important and hitherto unobserved. Having noted this caveat we will reflect on the quantity of observation visits conducted during this study. We will see later that the observation vantage points used in this study did not achieve sufficient purchase on the matter of organisational culture. More or longer observations from the same vantage points would not have remedied this. Aided by earlier discussions with service provider coresearchers, the non-clinical health services researchers became confident about their evaluations of team-level factors relating to safety culture. A total of 12 half-day visits (six per researcher) was sufficient for confident scoring. Scoring after fewer visits, say four or five per researcher, would have been possible once researchers had gained confidence in observing these particular clinical environments. However, reduction in the number of visits reduces the sampling of days of the week and times of the day. We believe that it is important to collect data in the week (Monday–Friday) and at weekends, during ‘office hours’ (9 am to 5 pm) and at other times of the day and night, at the beginning, middle and ends of shifts (noting that different staff groups have different shift patterns).

Theoretically, there was a risk of a Hawthorne effect: staff providing care more safely than when unobserved. However, clinicians’ work is routinely observed by others, including patients and their visitors; official visitors to the hospital; staff from other hospital departments; junior doctors on rotation; agency and locum staff; and patients, visitors, students and colleagues. Our experience was that busy clinicians quickly appear unaware of or unperturbed by researchers conducting observations. We looked for but saw no evidence of staff concern about being observed, although at one site there was concern that we could overhear discussion of named patients. Monahan and Fisher223 point out that, even if there is an undetected Hawthorne effect, this does not render observations worthless: if members of the clinical department ‘stage a performance’ then this reveals their perceptions of the performance that ought to occur. If the staged performance demonstrates bad practice then that provides quite robust evidence of low safety standards.

The tender specification (see Appendix 1) acknowledged the possibility that it may not prove possible to obtain distinct observation-based evaluations of organisational and safety culture and this proved to be the case (see Chapter 9, Limitations of this study). It was possible to assess three categories of organisation and work environment factors and five categories of safety-related teamwork factors (see Appendix 6). We were able to gather evidence of the adequacy of staffing, premises and equipment, the quality of support offered by administrators, senior managers and other related teams, and key aspects of teamworking: task management, decision-making, cooperation, coordination, leadership, communication, mutual performance monitoring, back-up behaviour, adaptability/flexibility and team/collective orientation. 11,12 We could observe the ‘active sharing and updating of knowledge, enabling risks to be collective and progressively monitored’;224 the maintenance of situational awareness;34,74,75 and other safety-related behaviours. The observations made during this study were consistently validated by staff when researchers reported back to departments, suggesting that the method developed is effective in capturing core team characteristics.

1. Objective 4: To obtain criterion-based measurements for the quality of care at each site.

This was achieved using a retrospective criterion-based audit of case notes, conducted by non-clinical auditors. Three common conditions, which had national guidelines for care, but had not been the subject of recent national audits or quality improvement campaigns, were selected. The conditions audited in delivery suites were normal labour and delivery, ECS and care following detection of grade 2 or grade 3 MSL. In EDs the audited conditions were ACS, ASA and FNoF. For each condition between three and five evidence-based audit standards were selected (see Chapter 3, Strand C: criterion-based assessment of quality of care). Some standards concerned treatment interventions, whereas others concerned recording important observations.

The study design planned sequential audits at the eight participating hospitals over a 2-year period, auditing consecutive cases and working backwards for no more than 6 months from the date observers first became noticeable in the clinical department. This plan addressed several points of general methodological interest. Sequential rather than parallel data collection at eight research sites was planned for efficiency; the study was feasible with two half-time research fellows. The selected time period simultaneously addressed concerns about a possible Hawthorne effect and the need to treat the survey, observation and audit data as contemporaneous in subsequent data analysis. However, sequential data collection requires vigilance with respect to the publication of revised care standards. It was important to conduct the case note review using the relevant standards as they were at the time the clinical records were made. Using personal research networks and seeking advice from the study’s senior clinical advisors, the study team had taken care to learn as much as possible about any likely revisions to standards before the final selection was made. Happily, only 1 of the 22 audit standards (ASA, see Chapter 7, Acute severe asthma) changed unexpectedly182,207 and particular care was taken with auditing this condition. Vigilance with respect to changes in national care standards is an important feature of audit-based research.

The audit results exhibited substantial variation between standards. For example, there was a very high rate of compliance with standard ECS4, ‘Women having an emergency caesarean section should be offered regional anaesthesia (spinal or epidural)’ (range 92–100%), whereas compliance with standard ECS1, ‘Documentary evidence of consultant obstetrician involvement in the decision to ECS’, was lower and more variable (range 55–84%). There was also substantial variation in compliance within audit standards; for example, compliance with the standard ACS3 (below) varied between 7% and 61% at different research sites.

If patient in pain, times from arrival to the administration of pain relief:

• in cases of severe pain, 50% in 20 minutes, 75% in 30 minutes, 98% in 60 minutes

• in cases of moderate pain, 75% in 30 minutes, 98% in 60 minutes.

Both between-standard and within-standard variation provide challenges for audit-based research. Safety-related markers of the quality of care are not all equally important: poor scores for certain markers may have disproportionately serious consequences. Therefore, it is important to keep variations in performance within individual standards visible in the reporting of results, as we did in Chapter 7. Nevertheless, some situations require an aggregate view of the quality of care, which was the case in this study. This is achieved by auditing a ‘basket’ of care standards so that the aggregation of results smooths out unusual results for one or two standards to avoid results which are simply artefacts of the care standard or narrow range of standards chosen. The greater the number of standards, the greater the smoothing: Hutchinson and colleagues,156 for example, used much larger numbers of standards than this study.

Power calculations for this study showed that nine audit standards were required for 80% power at 5% significance,174 hence the selection of a minimum of three standards for each of three common conditions for each type of site. Where there was clinical concern that certain standards may not differentiate between departments because of near-universal compliance or very low levels of compliance, additional audit standards were identified to mitigate the problems. The standards selected for this study were selected from those identified as having strong underpinning evidence, each was considered clinically important and they were equally weighted in analyses (Chapter 3, Strand C: criterion-based assessment of quality of care).

Care standards requiring the recording of repeated observations generally showed low levels of compliance. In some situations, such as FNoF a single follow-up observation FNoF5, ‘If in pain, evidence of re-evaluation of pain (90% of those with severe pain re-evaluated within 30 minutes; 75% of those in moderate pain within 60 minutes)’, exhibited compliance levels in the range 0–4%. It is not known whether this represents poor care or poor recording of care. For care standards such as MSL4 (baby assessment should include at 1 and 2 hours and then 2-hourly for 12 hours: general wellbeing; chest movements and nasal flare; skin colour including perfusion; feeding; muscle tone; temperature; heart rate; and respiratory rate) compliance was poor, in this case in the range 0–6%. A similar situation was found with the care standards for repeated observations during normal labour and delivery. This raises questions about the national care standards themselves. Possible explanations include that the care standards may not have widespread clinical support owing to considered professional judgement or lack of knowledge of the detail of the standard; staff may find the workload of making and/or recording multiple and repeated observations too high, signalling a need to prioritise those which are most important. Dixon-Woods and colleagues125 discuss how rules in practice are so numerous that they quickly exceed individuals’ ability to act on them. Staff then have to choose which rules to ignore, and may also ignore others without realising that they are doing so: partial compliance becomes the accepted ‘normal–illegal’ state. 225

The recording of multiple and repeat observations was also influenced by site-specific record forms. The structure of these enhanced or limited the recording of pertinent clinical information, depending on the design of the form. It is possible that structured record forms also prompt the completion of included clinical actions or, by the same token, could direct attention away from other clinical actions.

Some audit-based research, including this study, seeks to differentiate research sites by audited performance against standards. Where the national care standards failed to differentiate between departments, because compliance was very high or very low overall, researchers can still discern variation in performance. In the cases where audit standards detect uniformly low levels of compliance, it may be useful in future to devise a modified way of measuring performance. For example, in the case of ND (all three standards), this could be done by selecting a smaller number of observations agreed to be priorities by an expert panel, although this introduces the risk of different studies selecting different subsets of observations and compromising the comparability of study findings. In the case of treatment or tests that should happen within a specified time (ACS2, ASA2 and ASA3), the more inclusive measure could include moderately but not very late performance of these tasks: again, consistency between studies in the definition of ‘late’ would facilitate comparison of results. When standards failed to discriminate between sites because they were met in the majority of cases at all sites, such as the offer of local anaesthesia for ECS, it is not possible to increase discriminatory power in the same way and a replacement audit standard would need to be identified.

Audit standards such as MSL4, above, highlight the close-coupling or interdependent nature of clinical departments and teams,139,144 which we highlighted in Chapter 4, Close-coupling of departments and services. The MSL4 observations must begin in the DU but would normally be completed on a post-natal ward. Similarly, standard FNoF3,‘Radiography performed within 60 minutes of arrival in 75% of cases’, does not entirely lie within the control of the emergency department team. Audit-based research needs to address the complexity of overlapping responsibilities or flexible boundaries. Challenges may be less if the unit of analysis is a care pathway, which might be viewed as a control system,226 but sometimes team- or department (subsystem)-level evaluations are required and boundaries of responsibility and control must be considered. However, this is difficult, as Osman226 (pp. 122–3) cautions: ‘Reasoning about a network of decisions with respect to a single goal is imperative in order to monitor the behaviour of the system.’ But dynamic information exchanges across social and technical networks, which form and link subsystems, generate multiple decisions that will vary in extent and predictability in their influence on the particular goal being investigated in the evaluation of a particular subsystem. We may not be able to fully disentangle the effects of contributions (and delayed or missed contributions) from closely coupled departments or teams when auditing the performance of a particular department or team in a wider system.

In this study non-clinical auditors were preferred because earlier research found this reduced hindsight bias,186 and more recent research has found that non-clinical auditors are no less reliable than clinical. 156,160 This was by far the most challenging aspect of this study. The current climate in research governance places such severe restrictions on audits by non-clinical auditors that future studies using this methodologically sound and low-risk research approach seem unlikely. Extremely busy staff within the care team do not have the capacity or inclination to provide anonymised data for research studies. It is a moot point whether the small number of trust-based central audit clerks are members of the care team as defined by the NIGB, although trusts participating in this study all viewed access to medical records by these staff as unproblematic. By the same token, most trusts participating in this study believed that, as researchers were in any case already screened with all the employment checks conducted for trust staff and, after clearance, awarded honorary contracts, they should be able to conduct audit-based research. Recent guidance196 regarding increased access to patient data in health services research may ease some of the difficulties experienced during this study. However, the emphasis on restricting access to clinical records remains. Furthermore, the main focus of current policy223 and initial developments is on increasing and speeding recruitment to clinical trials; improvements for audit-based research may take longer to achieve.

A separate trend that occurred over the duration of this study was improved arrangements for reimbursing trusts for research support costs, which has eased the participation of NHS staff in research studies, but, once again, the emphasis is on work with consented patients for clinical studies. The extraction of anonymous audit data is still poorly supported. When this study sought advice from the NIGB the advice indicated that trust provision of anonymised data was considered feasible and appropriate; a second approach would be for trusts to send a letter to each patient requesting individual consent before allowing researchers to audit records. This study had already established that members of the care team could not provide anonymised data. The individual consent route was not feasible for this study because electronic records are not yet sufficiently well developed to act as anything more than a guide to identify cases that meet the specified audit criteria. Paper and electronic case notes have to be retrieved and read to establish eligibility for inclusion in the audit before letters can be targeted appropriately; in some cases, the preliminary screening alone equates to the level of work required to audit the case notes. This study was completed following lengthy efforts by trusts to identify permanent or temporary clerical staff to extract audit data. In one trust where all such efforts failed, a health-care assistant was engaged to complete the task. Arguably, the use of carefully screened non-local auditors is more ethical than the use of local non-clinical auditors, since the non-local auditors are far less likely to know any of the patients or staff personally.

A second methodological difficulty and ethical concern with excessive restrictions on case note review is that data will be disproportionately lost from members of the most mobile populations, including travellers, refugees and people in temporary accommodation; people with poor literacy or poor understanding, whether due to unidentified or unmet translation needs, general reading difficulties or cognitive impairment; and people whose deteriorating health has precipitated a move from the home they had at the time of their admission. The cumulative effect of these losses is to bias results, and there is already concern that the experiences of minority and vulnerable groups are insufficiently captured by current NHS data collection. 227

1. Objective 5: To compare levels of agreement between the questionnaire and holistic measurements of climate/culture.

This objective was linked to two hypotheses:

1. H1a: There will be a strong correlation and good agreement between questionnaire-based and holistic evaluations of organisational culture;

2. H1b: There will be a strong correlation and good agreement between questionnaire-based and holistic evaluations of safety culture.

These comparisons were reported in Chapter 8, and Table 40 reproduces the relevant rows from Table 38 with additional information on the closeness of agreement, extracted from Table 39. For the three comparisons between survey-based assessments of climate and observation-based assessments of culture low levels of correlation were found, indicating that there is no linear relationship between these measures.

Agreement addresses the closeness of two measurements made on the same scale. In this study, all summary scores were scaled to the standard normal distribution before comparisons were made. Table 40 shows reasonably good agreement between survey-based organisational climate and observation-based organisational culture scores. The SD of differences between scores was 1.089 and the majority (56%) lay in the range ± 0.5 points. The Bland–Altman plot for this comparison, Figure 10, drew attention to some clustering of points. This was caused by lack of discrimination in the observation-based organisational culture measurement. Although these two measurements were mostly very similar, the survey-based organisational climate measure is preferred. This is because of the disadvantage of poor-to-moderate survey response rates being considered less serious than the limited facets of organisational culture that could be observed in this study (see objective 3 above) and concerns about lack of discrimination in the scoring of the observation-based organisational culture measure.

The prevalidated scales from the NHS annual staff survey, which contributed to this study’s summary organisational climate score, were also compared with observation-based organisational culture scores; nether exhibited better correlation or agreement than our summary measure.

Table 40 shows that good agreement was found between scores for this study’s summary measure of safety climate and observation-based teamwork culture scores (five factors) and also the observation-based holistic safety culture scores (three organisation and work environment factors and the five teamwork factors from the previous measure). The SDs of differences were similar, 1.082 and 1.059 points, the distribution of points on the Bland–Altman plots (Figures 13 and 14) showed no discernible patterns, and around 40% of differences between measurements were small (within ±0.5). This suggests that survey-based and observation-based assessments of safety climate/culture may be considered equivalent. Adequately trained and experienced non-clinical researchers, who made 12 visits to clinical departments at purposively sampled times to conduct semistructured observations lasting 4–6 hours, made assessments of safety culture in clinical departments at similar levels to the perceptions of people working in these clinical departments, as reported in responses to questionnaire items drawn from the NHS annual staff survey175 and Sexton and colleagues’25 teamwork and safety climate scales. Thus, observation-based assessments of safety culture using the framework and scoring system developed during this study might be a useful alternative to staff surveys of safety climate, particularly as survey response rates appear to be falling and concerns about social desirability bias have grown.

An additional use for the observation-based assessment framework and scoring system developed during this study would be to complement staff surveys: the two measurements may be similar for most sites, but Figure 14 shows one research site on the upper limit of agreement. Here, staff survey responses indicated a much poorer climate than observer evaluations of teamwork culture. This multimethod study was able to trace this result back through field notes and survey responses to discontent with management style, despite generally well-executed teamwork. Using the contrasting measures in this complementary way was not envisaged in the commissioning of this study, but is a strength of multimethod studies.

The prevalidated scales that contributed to this study’s summary survey-based safety climate measure were also separately compared with the observation-based assessments but the summary measure provided superior agreement.

1. Objective 6: To compare organisational culture with safety culture.

This objective was linked to the following hypotheses:

1. H2a: There will be a strong correlation and good agreement between questionnaire-based evaluations of organisational and safety culture;

2. H2b: There will be a strong correlation and good agreement between (quantified) holistic evaluations of organisational and safety culture.

Table 41 reproduces the relevant extracts from Tables 38 and 39. It shows that the survey-based measurements were strongly correlated (r = 0.845), meaning that a linear relationship exists between scores on these measures. Agreement between the scores on these measures was good, with 88% of differences lying in the range ±0.5 points. Clinical departments in which staff rated safety climate favourably also returned favourable organisational climate scores. Further research would be needed to examine whether one favourable assessment is antecedent to the other. Zohar and Luria142 examined this in the context of manufacturing plants. They found that organisation-level and workgroup-level climates were globally aligned (as was the case in this study). In the manufacturing context, multilevel modelling revealed that supervisory practices varied between workgroups, producing variations in safety behaviour, but organisational climate set limits on variations in supervisory behaviour and, therefore, on safety behaviour within workgroups. It is not known whether this finding would be replicated in health-care contexts.

The comparison between observation-based organisational and teamwork culture scores (rather than organisational and safety cultures scores) was made because it did not prove possible to develop an observation-based holistic safety culture measure that was distinct from the observation-based organisational measurement. The comparison between organisational and teamwork cultures found low correlation (r = 0.356) and a lower level of agreement than was found in the previous comparison between survey-based measures. The SD of differences for observation-based scores was 1.135 points (compare with 0.449 for survey-based scores), and only five (31%) differences were small (within ±0.5 points). As previously discussed, in this study, observation-based assessment of organisational culture was felt to be limited. We are more inclined to state that the methods of observation and scoring did not allow sufficient purchase on organisational culture than to state that the survey-based assessments produced different results from the observation-based assessments. Time-limited non-participant observation in clinical settings cannot adequately assess organisational culture. Staff surveys remain an inexpensive form of assessment, notwithstanding concerns about falling response rates and non-respondent bias. Continued investment in traditional ethnography would also support analytical purchase on organisational culture.

1. Objective 7: To compare culture measurements and criterion-based measurements of the quality of care.

This objective was linked to two hypotheses:

1. H3a: There will be a moderately strong correlation and reasonably good agreement between criterion-based measurements of the quality of care and, firstly, questionnaire-based and, secondly, holistic evaluations of organisational culture;

2. H3b: There will be strong correlations and good agreement between criterion-based measurements of the quality of care and, first, questionnaire-based and, secondly, holistic evaluations of safety culture.

Table 42 reproduces five rows from Table 38, with additional annotations reproduced from Table 39. These show the comparisons between assessments of climate or culture and assessment of the quality of care from the criterion-based audit. In each case correlations were low, less than ±0.2, or in one case moderate (r = –0.481) because two outliers had a disproportionate effect. This study did not find a linear relationship between assessments of climate or culture and the quality of care.

Reasonably good agreement was found between the quality of care, represented by standardised criterion-based audit scores, and the following assessments of climate and culture:

• survey-based organisational climate (SD differences 1.058, with six (38%) differences in range ±0.5)

• survey-based safety climate (SD of differences 0.992, with nine (56%) differences in range ±0.5)

• observation-based holistic safety culture (SD differences 1.314, and five (31%) differences in range ±0.5)

• observation-based teamwork culture (SD differences 1.241, with seven (44%) differences in range ±0.5).

It can be seen that the closest agreement was with the summary survey-based safety climate measure developed for this study, which had the lowest SD of differences (0.992), giving rise to the narrowest limits of agreement (0 ± 1.98), and also having the highest percentage (56%) of small differences (0 ± 0.5 points). Survey-based evaluation of safety climate provides the closest agreement with audit-based assessment of the quality of care. However, in studies in which an observation-based assessment is preferred, the observation-based teamwork culture scores provided closer agreement than the observation-based holistic safety culture scores (SD of differences 1.241 compared with 1.314, and 44% small differences compared with 31%). There is still much that is unknown about the link between clinical performance and measurements of safety climate or teamwork culture. These measurements may complement and illuminate some audit results, but audits of performance must continue.

This study found poor agreement between observation-based organisational culture scores and criterion-based audit scores. This was predominantly due to lack of discrimination in the observation-based measurement of organisational culture.

1. Objective 8: To collect data such that, where sufficient respondents exist within a category to protect anonymity, data can be explored by: stakeholder group (e.g. managers, midwives, nurses, doctors, allied health professionals, support staff) and level (e.g. management responsibility).

The demographic profile of respondents broadly reflected the wider NHS workforce with respect to gender and ethnicity. Most (89%) staff survey participants held clinical roles and 37% managed other staff.

Responses to prevalidated scales within the staff survey were investigated using multilevel modelling. A range of individual-level factors were found to mediate scores on one or more scales, including being a manager, gender, age, years of employment with the trust, professional group, ethnicity and participation in certain types of CPD (particularly having a mentor, but also online training, updating skills and knowledge, shadowing someone or receiving on-the-job training). This expands the range of factors investigated in previous studies. 217,228 The details of the mediating effects can be found in Chapter 5, Factors influencing the indicators of organisational climate and Factors influencing the indicators of safety climate, and a summary is provided in Table 43.

At site level, the type of service (DU or ED) mediated error wisdom scores (i.e. the effectiveness of systems for reporting and handling errors; DUs perceived higher organisational error wisdom) and response rates mediated line management scores (departments with above-average response rates returned lower line management scores). There was no statistically significant residual variation at hospital level, but there was considerable residual variation at the level of respondents nested within research sites after allowing for the effects of individual-level characteristics. This indicates that scores on the scales selected to elicit perceptions of aspects of organisational climate and safety climate predominantly varied at the level of the clinical department, not at hospital level, reflecting the findings of other studies. 137,143

Our findings broadly support existing research that shows a more positive assessment of safety climate by managers than by clinical staff. 37,102,103,229,230 The teamwork climate25 results, but not other components of the survey-based safety climate measure, accord with other studies that found nurses’ assessments of aspects of safety climate more negative than doctors’ assessments. 106,218,228

Limitations of administrative systems

Hospitals’ administrative systems varied widely and were often not well suited for interrogation in relatively simple ways. For example, every research site struggled to produce a staff list for the staff survey. Locating records for the criterion-based audit varied in difficulty. At one ED, inconsistency of coding meant that some FNoF cases could not be differentiated from other fractures on the electronically generated list of cases, so records had to be manually sifted first. Similarly, most electronic lists did not contain the information needed to determine whether cases were eligible according to our criteria (see Table 1). One would not expect administrative systems to be tailored for research rather than clinical use, but some of the database coding deficiencies identified during this study were also sources of irritation and inefficiency during the routine work of clinical and administrative staff. Deficiencies that hampered the criterion-based audit for this study equally hamper trust-based monitoring for quality improvement.

Recruiting and deploying service user coresearchers

It was difficult to recruit service users interested in working as co-observers in clinical departments that were geographically spread across six strategic health authorities in England (see Chapter 3, Selecting the sample of research sites, and Chapter 4, Recruiting service user and service provider coresearchers): only two were recruited, and one withdrew before completing research governance screening. Recruiting local service user coresearchers was the logical alternative, but this study’s sequential recruitment of research sites and tight timetable for data collection did not allow sufficient time for interested individuals to be identified and complete trust-based research governance screening. At four of the hospitals included in this study, research governance screening for the service-user coresearcher could not be completed before the data collection period ended, preventing her participation. The service user coresearcher did not attend meetings of the project steering group, initially due to late recruitment.

Tensions and challenges in quantifying observational data

The process of creating the required observation scoring mechanism (see Chapter 4, Quantifying semistructured observations, and Appendix 6) was challenging, and a major part of the methodological innovation in this study: it therefore warrants extended discussion. In this section, we first review the process that led us to the adopted scoring system and then discuss the rationale for this method.

Scoring did not begin until observation data were available from half of the study sites to ensure that no site had undue influence on the emergent scoring framework. Initially, we explored the possibility of awarding separate scores to items on the observation data collection prompt sheet (see Appendix 5). We chose a scoring system modelled on Likert scales: the highest score for strong and consistent evidence for the attribute of interest; the lowest for weak, rare or non-existent evidence. Such a scale has been used for team audit purposes for safety culture. 84 For each aspect of safety culture, we tried to score separately at the organisational and team levels (to mirror the survey data). This process helped us to understand which facets of safety culture would be amenable to data collection and scoring, and the nature of difficulties that would arise.

The research fellows carrying out the scoring found it difficult to reach judgements that they both trusted. The main reasons for this were researchers’ caution and differences between data sets, of four kinds:

• differences between sites, in current practice, facilities, etc.

• differences in what we were able to observe (see Chapter 4, Observable activities)

• differences between what was seen by different observers at the same site and

• differences in how each scorer categorised similar material.

For example, it was difficult to give convincing comparative scores to sites in relation to their use of the whiteboard when some sites had none, although the absence did not mean that teams did not communicate effectively.

Other possible sources of difference that when investigated did not prove to be matters of concern for this study were:

• differences between the composition of observation teams (see Chapter 4, Recruiting service user and service provider coresearchers)

• differences over time as observers became more experienced.

In fact, observations by service user and service provider observers were highly congruent with those of research fellows; and when research fellows later reviewed their earlier notes, they did not find significant differences from those made later.

Many of the agreed scores at this stage were unsatisfactory to the scorers, who felt them to be indicators of compromise rather than of consensus. A more parsimonious framework was sought, both as a route to facilitating consensus and to make any emergent scheme more accessible and economical for future users. We experimented with grouping aspects of safety culture from the data collection prompt sheet (see Appendix 5) into a smaller number of domains, under two overarching headings: ‘organisation and work environment factors’ and ‘team factors’. After several iterations and trials of scoring, the scoring system used (see Appendix 6) was agreed. The use of broader domains enabled the scorers to look at subsets of data clustering round domains rather than try to interpret small pieces of data separated from their context. For example, communication as a whole was scored rather than the use of the whiteboard in isolation. Using this method, each scorer found it easier to reach an individual judgement in which he or she had confidence, and together he or she found it quicker to reach consensus in those instances when he or she disagreed. This continued to be the case when applied to data from later sites. It is worth noting that the results were sometimes counterintuitive: systematic reference to the notes in these cases generated domain scores different from ‘off-the-top-of-the-head’ scores. Indeed, scorers sometimes felt the need to double-check the data records to ensure that a general impression had been mistaken. Such counterintuitive results strengthened our confidence that our scores were not impressionistic but grounded in recorded data.

Using a smaller number of domains also made agreement between scorers easier to achieve. First, the use of eight, rather than 32, categories reduced disagreement by removing the likelihood of disagreements about definition. Whereas scorers would debate the boundaries between two or more of the 32 categories, this did not happen when using eight. Second, it became much easier to reach consensus when the categories were more inclusive; for example, disagreements at one site about ‘uniprofessional respect and collaboration’ reflected different exposure of observers to the dominating behaviour of a single consultant. These were easier to resolve when this category was merged with others into the more inclusive ‘respect/warmth/collegiality’, because the scorers agreed about the other components of that domain. Disagreements were often to do with either scorer needing the other’s help to see that particularly vivid experiences should not be given undue influence in determining the overall score. This was an important benefit of engaging two observers to work in partnership, in preference to extending the observation time of a single observer.

However, as already noted (see Chapter 6, Observation-based evaluations of organisational, teamwork and safety cultures), the four-level scale, described in Chapter 4, Quantifying semistructured observations, resulted in several tied scores, indicating that the scheme provided insufficient differentiation between research sites. Increasing sensitivity by increasing the number of levels in the scoring system, or increasing the number of domains scored, was investigated. After piloting, both approaches were felt to be reducing reliability to a greater extent than they were improving sensitivity. We prioritised reliable scoring for this study. Further research is needed to find ways to improve the sensitivity of scoring observations.

The evolution of the scoring framework in Appendix 6 led to the exclusion of a number of potential items (see Appendix 5). These related to material that relied more on report (during conversation with staff) than on observation. The main problem with reported data was the variability and limited number of opportunities for conversations to illuminate and contextualise observations. Relying on a small number of opportunistic conversations did not allow observers to make informed judgements about scoring and the presence and nature of any bias in conversational reports. A more extensive ethnographic approach (or another multimethod approach) to data collection would have been required to retain the excluded facets of safety culture, such as action outcomes from incident reports.

Service user and service provider observers did not take part in the development of the scoring system or the final scoring: the time commitment would have been too great. However, all observers from each site met to discuss their findings in detail, and to ensure that any differences in observations and/or judgements were clearly explained and recorded in field notes that contributed to the scoring process.

For the final scoring, two research fellows (SA and MR) separately scored observations made at every site then discussed any differences and agreed joint scores. Owing to staff changes during the project (see Chapter 4, Impact of staff changes in research team), these two research fellows did not collect data at sites 1 and 2. At the scoring phase, EJB, who collected data at these sites, helped to resolve uncertainties about the scoring of two (from nine) scores for site 1, based on field notes relating to sites 1 and 2, which were written by EJB and NM (see Table 27). She also reviewed and agreed with the remaining 14 scores for sites 1 and 2. The difficulty in confidently assigning two scores highlighted that it may be important for observers to undertake scoring or for scorers to have recourse to discussion with observers. The ability to make confident and verified judgements based on other observers’ field notes for 88% of scores indicated that, with sufficiently detailed or well-structured field notes, experienced researchers could score others’ observations. This would mitigate the effects of staff changes, or the use of local researchers on multiple sites.

There were several reasons for quantification in this study. Most basically, the commissioning brief required not only that a ‘holistic’ method of data collection was used, comprising largely observation, but also that the results be available for statistical analysis in order to test for correlations with survey scores: a positivist view of triangulation at variance with the multiple forms and several purposes of triangulation in the qualitative and multiple methods research literatures. However, despite some reservations stemming from our qualitative and multiple methods research backgrounds, in choosing to accept the commission we gave tacit endorsement to quantification as a valid method in this context.

Qualms about quantifying observations that are initially recorded as continuous field notes may arise from viewing narratives and grading as epistemologically incompatible. However, the gap between the two is not necessarily as wide as is often perceived: an important proportion of quantitative data are in any case a quantification of attitudes and judgements, rather than a measurement of physical phenomena. For example, survey questions (including our own) frequently ask respondents to review their knowledge and observations and to reach a judgement that can be scored, for example by Likert scales, although how these judgements are reached is invisible to the researchers. In the case of our ‘observation’ method, we have attempted to be more transparent about the method whereby these judgements are reached, and this has required a detailed description of how we made observations and subsequently scored them. The first part of this description may resemble that provided for a purely qualitative study, but this does not alter the fact that the rationale for the observation was ultimately quantitative.

We chose to record field notes at the point of data collection rather than rely solely on a checklist that would yield easily quantifiable data, in order to minimise the risk that we failed to record data that later turned out to be important. Although the initial detailed checklist was based on indicators of organisational and safety culture drawn from the wider literature (see Chapter 3, Developing the observation prompt list) and insights gained in our previous work in DUs,34 and the method had been piloted in a local ED, we were, nonetheless, aware that we might encounter situations in which items might not be relevant and comprehensive in relation to a particular setting. An early example was the presence and use of a whiteboard at site 3, which was much more restricted than we had previously encountered. At site 3 computer screens, instead of a whiteboard, were the main mechanism for recording up-to-date patient data. Having made detailed notes of what we saw, informed by the prompt list (see Appendix 5), we were able to refer to field notes from this and other sites about other modes of information exchange and handover (informal, one-to-one, consultation of computers) to assist us in making comparative judgements. Had we scored a checklist in the field, we would have much more restricted information from which to formulate ways to cope with the unexpected. Similarly, by recording field notes, we made it easier for observers to discuss their, at times varying, interpretations of the same phenomena. For example, the research fellows initially differed in their assessment of the senior consultant at one site, one finding his leadership style authoritarian and hierarchical, the other noting his consistent willingness and availability to give support to junior staff. Each observer could check his or her own notes against the other’s interpretation, and a consensus based on recorded data rather than on recall was quickly reached, that both impressions were in fact valid. In addition, by recording field notes, we made it possible to reconsider data in the light of any new perspectives from service user or professional observers, whom we were not able to recruit before site 6. Detailed field notes also mitigated the impact of staff changes.

For all these reasons, the recording of field notes was preferred; but at no point did we forget that the purpose of observation was to generate quantitative data relating to safety culture. Indeed, team members experienced some frustration in the early stages of the study in not being able to finalise the scoring method until there were sufficient data to test it.

Multiple delays to the data collection timetable

Even for the experienced research team conducting this study it was unusually difficult to accommodate the number and variety of delays to the data collection timetable that were encountered. These were due to:

• protracted decisions by hospitals about whether or not to participate in the study

• very time-consuming processes for research governance at hospital level

• variations in the time taken by trust-based auditors.

Of these, the second was by far the most important.

Patient consent for notes audit

The NIGB requirement that case notes can be audited only by the care team unless individual patients consent was a significant challenge to us. If this view is applied consistently from now on, the viability of the use of case note audit by non-hospital-based research teams is much reduced, given the likelihood of low response rates to letters requesting permission and the reluctance of hospitals to disclose names and addresses. In our experience, hospitals did not find it easy to provide their own staff for external research purposes, even though this work would be fully reimbursed. During the period of this study, arrangements for reimbursement of trusts’ research costs improved (in value and simplicity). In other research we conducted over the same time period we noticed increased enthusiasm for research participation, which seemed to parallel the improved system for research support costs. However, our audit-based research seeking non-clinical auditors remained difficult to support. Perhaps there is a shortage of non-clinical auditors, or perhaps audit work is not attractive to non-clinical staff. It is to be hoped that recent guidance on improving access to clinical data for health services research196 results in changes that better support audit-based research. However, the initial focus is on speeding recruitment to clinical trials, rather than health services research. 222

Introduction

Management researchers have long been concerned with exploring the role that organisational culture may play in influencing the efficiency and effectiveness of organisational performance. More recently, this research has begun to be adapted and applied to a health-care context. 2 There is increasing interest in the idea that one way of improving health-care performance factors such as quality, efficiency and patient safety could be through influencing professional and organisational culture. 2,3

Background

While business researchers seek to measure culture (and all NHS staff in England and Wales take part in a yearly staff attitude survey), many top business people, such as Harvey Jones and Archie Norman, believe they can perceive the culture of an organisation within a short period of time. In the more specific context of patient safety, the importance of creating a ‘safety culture’ is underscored in two key documents of the patient safety movement. 1,4 These reports suggest that such an outcome could be achieved by moving from a culture of blame to one of learning from mistakes. Some preliminary work has already been attempted to correlate the construct of ‘organisational culture’ with quality and performance. However, empirical evidence in support of a link between organisational culture and clinical performance (i.e. to test criterion validity) is weak. 5,6 Scott et al. 6 reviewed the evidence regarding organisational culture and health-care performance and identified several methodological difficulties. One problem is the fact that the studies that have compared culture measurements with criterion-based outcome or clinical standards are not only limited in number, but also in size. In order to test the validity of culture as a marker for quality/safety, it is necessary to have access to reliable measurements of both quality and culture.

As far as measuring quality of the process of clinical care is concerned, some steps have been taken to develop appropriate criterion-based indicators. 7,8 For example, Eric Thomas9 has reviewed different methods to measure safety and Lilford and colleagues10 have analysed their strengths and weaknesses and have articulated the importance of measuring error rates in terms of opportunity for error rather than patients treated. In addition, the NHS Research Methodology Programme is currently sponsoring a large study into the measurement of error/quality11 on behalf of the Patient Safety Research Programme and the Health Care Commission. The challenge now is to find out how safety culture correlates with criterion based measurements of the quality of clinical care. For the purpose of this study, an error is any violation of a clinical standard, whether an act of omission or commission. As far as culture is concerned, this can be measured at a general level, for example using the Organisational Culture Instrument currently used by the Health Care Commission. 12 More specific constructs of culture can also be measured, for example safety culture.