Development of a social inclusion index to capture subjective and objective life domains (Phase II): psychometric development study

Social exclusion and inclusion

Social exclusion is a term that was not widely used in the UK until 1997, when the Labour government established a coordinating policy body called the Social Exclusion Unit,1 which completed a major study of social exclusion and mental illness2 among other things.

The socially excluded have been defined as ‘Those people who do not have the means, material or otherwise, to participate in social, economic political and cultural life’. 3

Social exclusion focuses primarily on relational issues – inadequate social participation, lack of social integration and lack of power. 4 According to Berman and Phillips,5 social exclusion manifests itself at both the national and community level. Inclusion in society is normatively defined, for example as citizenship – having a job, home or financial security according to the norms of society. It also includes being part of, and identifying with, a community. In our heterogeneous society, community may be a more relevant measure of ‘inclusion’ than a national identity, which may be more amorphous.

Researchers in the poverty tradition in Europe made the distinction between ‘poverty’, which they define as a lack of resources, and ‘exclusion’, which is more comprehensive. It has been suggested that social exclusion should be defined in terms of the failure of one or more of the following four systems of ‘integration’:

• the democratic and legal system, which promotes civic integration

• the labour market, which promotes economic integration

• the welfare state system, which promotes social integration

• the family and community system, which promotes interpersonal integration.

One’s sense of belonging in society depends on all four systems. Civic integration means being an equal citizen in a democratic system. Economic integration means having a job, having a valued economic function, being able to pay your way. Social integration means being able to avail oneself of the social services provided by the state. Interpersonal integration means having family and friends, neighbours and social networks to provide care and companionship and moral support when these are needed. All four systems are therefore important. In a way, the four systems are complementary, in that when one or two are weak the others need to be strong; the worst off are those for whom all systems have failed. 6

Social exclusion can also be conceived of as a denial, or non-realisation, of citizenship rights: civil, political and social. The four major social systems referred to above should, when working effectively, guarantee full citizenship. Steps towards the creation of social inclusion will therefore involve ensuring that the systems operate effectively, preventing the exclusion of individuals and communities, with full citizenship as the result.

Social inclusion has been defined in the European Union (EU)7 as ‘A process which ensures that those at risk of poverty and social exclusion gain the opportunities and resources necessary to participate fully in economic, social and cultural life and to enjoy a standard of living and well-being that is considered normal in the society in which they live. It ensures that they have greater participation in decision-making which affects their lives and access to their fundamental rights (as defined in the Charter of Fundamental Rights of the European Union)’. The aim of the social inclusion policy in the EU is ‘to prevent and eradicate poverty and exclusion and promote the integration and participation of all into economic and social life’. 8 In 2008, Mental Health Europe produced a descriptive report from 27 member states (including Scotland and Ireland, but not England or Wales) on the outcome of its work programme on social inclusion for people with mental health problems. 9

As part of a literature review examining the links between social exclusion and mental ill health, which was undertaken at the Centre for Analysis of Social Exclusion and the Personal Social Services Research Unit (PSSRU) [London School of Economics (LSE)],10 one of the authors (TB) identified two broad schools of thought. The first may be called a rights-based approach in which social exclusion reflects the deprivation of rights as a member or a citizen of a particular group, community, society or country. This approach is strongly associated with the international literature on social inclusion. 11 The second approach starts from the assumption that social inclusion is the opportunity to participate in key functions or activities of the society in question. This approach is a development of the traditional concerns of social science and especially social policy: measuring poverty and multiple deprivation. 12,13 Table 1 combines Commins’6 forms of integration with these two broad approaches, which, it could be argued, are broadly similar to the concepts of ‘demos’ and ‘ethnos’. 5,14,15

Rights-based conceptions of social inclusion may be particularly important in the context of mental health, as a denial of rights and/or access to the means to realise entitlements has historically been a feature of the treatment of people with mental illness. Conceptions of social inclusion based on participation are also important, however, especially where comparisons with the general population are sought. Some authors attempt to integrate the two approaches. For example, Room,4 Abrahamson,16 Kronauer17 and Littlewood18 argue that social exclusion is the reinforcement of accumulated social disadvantages through the denial of civil, social and economic rights. Reimer19 proposes that social inclusion and exclusion processes are rooted in four types of social relations:

• market (exchange and barter)

• bureaucratic (rational–legal)

• associative (common interest)

• communal (complex reciprocity and shared identity).

As well as there being at least four systems of integration, there are also several levels at which they can be addressed:

• individual level

• family and close networks – perhaps the household level

• local community/local employer/local leisure activity and availability

• government policies and initiatives to promote personal and community integration; responses of wider society.

According to Burchardt et al. ,20 there are four aspects of social exclusion:

• consumption, where individuals do not have the capacity to purchase goods and services

• production, where individuals are unable to find employment

• involvement in local and national politics and organisations

• social interaction and family support.

Social inclusion is widely agreed to be:

• relative to a given society (place and time)

• multidimensional (whether those dimensions are conceived in terms of rights or key activities)

• dynamic (because inclusion is a process rather than a state)

• multilayered (in the sense that its causes operate at individual, familial, communal, societal and even global levels).

On the relationship between social inclusion and social exclusion, the point has been made that a person can be included in smaller, closer family or peer groups, yet at the same time excluded from the mainstream, or vice versa.

A further important distinction is between ‘risk factors’ for social exclusion and indicators of social exclusion (or inclusion) itself. For example, having a Pakistani or Bangladeshi background, lone parenthood, and chronic ill health are all risk factors for social exclusion but are not sensibly thought of as dimensions of social exclusion itself: they are indicators neither of rights nor of participation.

Critics of the social inclusion concept

Spandler21 has contested the idea that inclusion is always a desirable entity and has pointed out that the promotion of inclusion assumes that the quality of mainstream society is not only desirable, but also unproblematic. Zissi et al. 22 also question whether or not what they call the ‘mainstream’ notion of social constructs is always beneficial. They found that small rural communities which were advantaged by having a strong sense of belonging were nevertheless disadvantaged by not having strong external links with civic power agents. Osborne et al. 23 found that women with families reported difficulties in combining group involvement with family responsibilities. Stress produced in the process of negotiating social interaction within groups was having a negative impact on their mental well-being.

Social quality

For some people, the concepts of social exclusion and inclusion are still too narrow as a basis for the development of effective local and national social policies. 5 Beck et al. 24 have proposed an alternative overarching conceptual framework of ‘social quality’, defined as: ‘The extent to which citizens are able to participate in the social and economic life of their communities, under conditions which enhance their well-being and individual potential’. 24

This concept of ‘social quality’ is similar to our conception of quality of life (QoL) assessment in mental health,25 in that it encompasses both objective and subjective interpretations. Social quality has at least four overlapping elements that bear a close resemblance to the life domains of our QoL assessments. They are:

• social–economic security (protection against unemployment, poverty, ill health and other material deprivations)

• social inclusion/exclusion (equal access to supportive infrastructures, labour conditions and collective goods)

• social cohesion/anomie (the availability of social networks, equal access to services)

• empowerment/disempowerment (enabling people to develop their full potential in social economic, political and cultural processes).

Vogel’s26 taxonomy of social quality is virtually the same as the life domains covered by our QoL measures, i.e. health, education, work, income, housing, safety, social attachment, leisure. 25

Finally, there is considerable debate about the perception of the term ‘social inclusion’ itself and whether a subjective perception of exclusion is necessary or sufficient to identify an underlying state of social exclusion. 27

Information about an individual’s subjective state is clearly important in any evaluation but especially where low self-esteem, experience of discrimination, and long-term deprivation are prevalent; in these circumstances, individuals’ expectations may be so reduced that subjective assessments of well-being should be treated with caution,28,29 although Evans and Huxley30 have shown that most of the time deterioration in material well-being is accompanied by a decline in subjective well-being (SWB) and vice versa in general population and mentally disordered groups.

Where counterintuitive findings apply, for instance in those with severe mental illness (SMI), subjects are more likely to make aspirational ratings rather than resigned ones (i.e. where resignation is denoted by high subjective QoL rating, when adverse circumstances are experienced). Lelkes31 found essentially the same results based on large survey data samples.

Related concepts and their measurement

It may be worthwhile taking some time to examine the question of which other concepts or constructs abut or overlap with social inclusion, and from which concepts it helpfully can be distinguished. Clear candidates on the basis of our concept mapping groups are citizenship, social capital, social participation, social cohesion, social support, community perceptions and QoL. A number of papers usefully address some of these inter-relationships and there have been some valiant and worthwhile attempts to incorporate them into a single model. 32

Citizenship

Stewart33 distinguishes state citizenship from democratic citizenship. Community citizenship refers to the possession by members of a community of a range of social and cultural rights and responsibilities, by virtue of their membership of that community and as a distinct element of their national citizenship rights. Roche34 suggests that citizenship can be seen as the core of what it is that social exclusion processes exclude people from, and the core of what social inclusion policies promise to include people in. Some writers cast doubt on the emancipatory promise of citizenship in the mental health field,35 while others argue that citizenship promotes both mental health and participation. 36

Social cohesion

According to Forrest and Kearns,37 the domains of social cohesion relate to:

• common values and a civic culture

• social order and control

• social solidarity and reduction in wealth disparities

• social networks and social capital

• place attachment and identity.

Some definitions of social cohesion are close to that of social capital (see next section and also Stanley,38 who defines social cohesion as the willingness of members of a society to cooperate with each other in order to survive and prosper). Willingness to cooperate means members choose freely to form partnerships and have a reasonable chance of realising goals, because others are willing to cooperate and share the fruits of their endeavours equitably. Freidkin39 argues that social cohesion is the causal system that determines individuals’ membership attitudes and behaviours. Thus, social cohesion is a property of communities and groups, whereas ‘being socially included’ is an attribute that an individual may or may not have. It has been suggested that good mental health is rooted in social cohesion,40 and low social cohesion has been shown to be associated with poorer mental health. 41

Social capital

According to the Health Development Agency42 the components of social capital relate to:

• participation in the local community

• reciprocity

• feelings of trust and safety

• social connections, power

• community perception.

Nevertheless, community perception is regarded by some as a psychological concept in its own right;43,44 participation is a key component of social inclusion according to many authors,29,45 etc.

Forrest and Kearns37 consider the key components of social capital as:

• empowerment

• participation

• associational activity

• supporting networks and reciprocity

• collective norms and values

• trust

• safety

• belonging.

Only participation and associational activity are widely regarded as social inclusion components.

The most widely cited definitions of social capital come from the Harvard political scientist Robert Putnam,46 who defines social capital as ‘a set of horizontal associations among those who have an effect on a community, and these can take the form of networks of civic engagement’ and ‘features of social organizations such as networks, norms and truths that facilitate coordination and cooperation for mutual benefit’.

Bourdieu47 understands social capital to be a process of deliberately constructing sociability in order to acquire the benefits of being part of a group. That is, social networks are not a natural given and must be constructed through investment strategies, which are grounded in the institutionalisation of group relations. The latter are useable as a source of other benefits. Bourdieu47 suggests that social capital comprises two elements:

• the social relationship that enables individuals to gain access to resources possessed by their associates

• the amount and quality of those resources.

Social capital is a multidimensional concept that encompasses a number of theoretical distinctions. Grootaert and Van Bastelaer48 suggest that there are two types of social capital: structural and cognitive. Structural social capital facilitates information sharing, and collective action and decision-making through established roles, social networks and other social structures supplemented by rules, procedures and precedents. As such, it is a relatively objective and externally observable construct and relates to the participation element of social inclusion. In contrast, cognitive social capital refers to shared norms, values, trust, attitudes and beliefs. It is therefore a more subjective and intangible concept.

A further distinction is often made between ‘bonding’, ‘bridging’ and ‘vertical’ forms of social capital. Bonding social capital relies on strong ties between people. It is inward focused and characterised by homogeneity, loyalty and exclusivity. Bonding social capital has a significant downside, as a tightly knit homogeneous community might be one intolerant of individual diversity, asphyxiating to live in and exclusionary to outsiders. 49

Bridging social capital, in contrast, links diverse groups and people. It is characterised by weak ties, has an outward focus and is likely to foster social inclusion. It is commonly recognised that this form of social capital is useful for finding employment. 50 As employment is key to many conceptions of social inclusion,51 it can often mean the difference between ‘inclusion’ and ‘exclusion’ for some people.

‘Vertical’ social capital is often distinguished from ‘horizontal’ social capital by virtue of the connections being made within a hierarchical structure to government and other institutions, rather than within and between communities. Vertical social capital provides a community’s institutional integration and, together with bridging forms of social capital, equates to an inclusive and cohesive society. 52,53 Moreover, socially cohesive societies high in social capital are more likely to achieve the twin virtues of sustainable economies and human development, according to Stanley. 54 Thus, high social capital is likely to be associated with greater chances of social inclusion, but is not sufficient to achieve social inclusion. Social inclusion refers to a broader range of domains, including for example realisation of rights, labour market integration, and so on. Araya et al. 55 provide some evidence in support of an association between mental health and individual perceptions of social capital.

Social support

It has been said that the measures of social support are as varied as the number of investigators. Veiel and Baumann56 created a useful conceptual framework in which they distinguished everyday support from crisis support, instrumental support from psychological support, and subjective from objective appraisals of support. Numerous support measures have been reported; mainly in the fields of psychology and social psychology. 56,57 Social support concepts and measures have developed in three different strands:

• the extent to which individuals are attached to significant others as measured by their social ties, participation in organisations, contact with friends and family and/or the complexity of their social network (e.g. social embeddedness)

• the individual’s cognitive appraisal (e.g. perceived social support) of the availability and adequacy of support irrespective of the extent of the support

• the responses of others in the provision of emotional support, information, tangible care or material assistance. 58

So, as for social capital, social support is an important facilitator of social inclusion, but may not be sufficient to achieve it.

Social participation

There are a number of ways to define and measure social participation. There was considerable interest in measuring participation in 1950s USA. 59–62 Earlier Chapin63–65 developed a brief scale of participation, which really amounted to a measure of organisational membership. Parker’s66 elaboration of Chapin’s work was also confined to activities (committee or informal) within formal voluntary organisations, rather than any other type of participation, and this approach is probably rather limiting in the current context.

Of all the other concepts, social participation is perhaps the closest to social inclusion, as most concepts of inclusion incorporate it as a key component. Larivière67 reviewed 77 measures of participation and concluded that there was a consensus that social participation is the realisation of activities. The most often used indicators include amount and frequency of activities. Van Brakel et al. 68 developed the social participation scale that we had planned to use in the present study, until we found a more acceptable and well-validated measure developed by Berry et al. 69,70

Although social participation has been shown to be related to positive mental health,71 adverse consequences of participation, particularly for women, could be a possibility. Osborne et al. 23 found that participation can reinforce gender inequality and have a potentially negative impact on women’s mental health.

Social networks

Social networks have been defined as the web of social relationships that surround an individual and the characteristics of those ties. 72–75 Burt76 has defined network models as describing ‘the structure of one or more networks of relations within a system of actors’. 76 Network characteristics cover:

• range or size (number of network members)

• density (the extent to which the members are connected to each other)

• boundedness (the degree to which they are defined on the basis of traditional group structures such as kin, work, neighbourhood)

• homogeneity (the extent to which individuals are similar to each other in a network).

Related to network structure, characteristics of individual ties include:

• frequency of contact (number of face-to-face contacts and/or contacts by phone or mail)

• multiplexity (the number of types of transactions).

Social support can affect a person’s health through different pathways: health behavioural, psychological and physiological pathways. 77

Eklund and Hansson78 argue that there is a reciprocal relationship between the social network and QoL of people with mental health problems. Social networks, as with social support and social capital, help to promote social inclusion but, in the absence of other mechanisms, are unlikely to be sufficient to attain social inclusion.

Quality of life

The Bristol Social Exclusion Matrix45 measure incorporates a section on QoL that includes such things as self-esteem and personal efficacy, personal development, etc. We would regard these rather as psychological or cognitive outcome indicators, improvement in which might be the target for service providers and service users. Generally, these are not accepted components of QoL measures, which are often health related and used as disease-specific treatment outcome indicators; generic QoL measures usually cover material or objective circumstances and SWB in very similar life domains to those identified in the social inclusion and exclusion literature. Berger-Schmitt and Noll32 incorporate QoL into their overarching model as the ultimate objective for social policies, including inclusion policies. We find their model coherent and appealing. So, social inclusion can be seen as a major contributory factor to an improved QoL in both individual life domains and QoL overall, with the caveat that for some people in some circumstances their conscious choice to exclude themselves could enhance their subjective QoL in a particular domain. The QoL of people with mental health problems has been extensively researched. 25,79

Well-being

In recent years, there has been a shift from a predominant concern with mental illness to an interest in the mental health and well-being of the whole population – public mental health. The World Health Organization80 has declared positive mental health to be the ‘foundation for well-being and effective functioning for both the individual and the community’ and defined it as a state ‘which allows individuals to realise their abilities, cope with the normal stresses of life, work productively and fruitfully, and make a contribution to their community’. This has been accompanied by a growing interest in developing indicators to measure different dimensions of positive mental health, to accompany indicators of psychiatric morbidity. 81 Affectometer 2 has been identified as a promising scale;81–84 and has been included in the Health Education Population Survey (HEPS) in Scotland. Affectometer 2 has been psychometrically validated in the UK, and a revised, shortened scale – the Warwick–Edinburgh Mental Well-being Scale (WEMWBS) – has been developed to better assess population positive mental health. 85 In the USA, Keyes86 confirmed that social integration, social coherence, social contribution, actualisation and social acceptance are the key components of social well-being. Well-being is found in this study to be an achievement facilitated by age and educational attainment. Like QoL, social inclusion should contribute to well-being, but is subject to the same caveat that, for some, being able to refuse inclusion may promote well-being. In principle, someone could be ‘well’ in the sense of having positive mental health, but nevertheless be socially excluded. Inclusion implies something positive about the relationship between the individual and society, while well-being is about the individual in him/herself. Wilhelm et al. 87 have recently called for a more precise operational definition of mental well-being.

Social inclusion

A review of the social inclusion literature88 led to the development of a framework to guide the selection of items and measures for a survey of social inclusion in northern Adelaide. It concluded that a broad suite of indices would be required to capture all of the relevant aspects of inclusion.

Morgan et al. 89 reviewed the literature on social exclusion at the same time as our Phase I study reviewed the literature on social inclusion. Both found that there was no single accepted definition of either concept, and that social participation (or the lack of it) was one of the core components. Both distinguished the social indicator approach from the individual subjective appraisal approach. Although it was suggested that measurement of social exclusion in the mental health literature was poor,89 our Phase I work went on to explore the dimensions of the concept of social inclusion in different social and patient groups before developing the present measure.

There is a relative dearth of research looking at the relationship between structural and subjective indicators of inclusion compared with the volume of publications and level of interest in the relationship between these variables and health status. The same is true of the literature on social capital. In some cases the argument is made for the exclusive use of the subjective approach,90 but in most cases the use of both subjective and objective indicators is advocated (Phase I report and commissioning brief).

An important consideration is the nature of the research design being used, and in particular the need to avoid the ‘ecological fallacy’. A considerable number of published projects use cross-sectional designs in which the subjective inclusion data come from individuals, but the contextual or structural variables come from aggregate social survey data, which are sometimes used to characterise descriptively the society from within which the individual respondents are drawn. As the health data also normally come from the individual, there is the possibility that the health and individual inclusion data are subject to information/reporting bias. For example, one such study found no relationship between contextual factors such as aggregate social trust and civic participation and subjective health measures when sociodemographic factors were controlled. 91 Nevertheless, the same study found that data about trusting and socially active individuals were associated with the same individual’s reporting of good health (in countries with high levels of social capital).

When considering these relationships in mentally unwell people there is also the possibility that the symptoms affect both inclusion as well as health status variables, producing a spuriously inflated positive or negative relationship.

In our approach to this issue we have used the same indicator questions (from surveys) but reported by the individual; we would, therefore, expect a somewhat more substantial relationship between O (contextual) and S (subjective) variables in our study than in studies using subjective data derived from individuals and contextual data derived from structural indicators.

Our updated literature search found considerably more work being undertaken on social capital than on social inclusion. This might be related to the newness of the concept compared with social inclusion, and to the promise of being able to account for health status and perhaps develop interventions (collective or individual) to improve health status. It is worth pointing out that social participation is of importance in both social capital and social inclusion, so the definitional problems involved in both may mean that some aspects of social inclusion are studied in social capital research and vice versa, depending of course on the particular definitions used. For example, in a study of social capital92 eight separate aspects were identified in a literature search and the first four of these were described as aspects of structural social capital, but are often used in measuring aspects of social inclusion and participation: ‘The first four capture structural aspects of social capital (contact with local family – referred to as family ties; contact with local friends – friendship ties; associational membership; and integration into the wider community operationalised by having social networks outside as well as inside the local area)’.

Clinical practitioners and health service researchers probably have a considerable interest in social participation, as this is likely to be a key component of recovery and rehabilitation, especially in the mental health field. In fact, there is evidence in the more recent literature that clinically based researchers are indeed focusing on social inclusion and participation, rather than social capital. 93–99

Social capital

A review of definitions of social capital within social psychiatry100 found that definitions are intertwined with the scales used to measure it, and that there was little consistency between studies in the literature. The review suggested that ‘it is clear that social capital is not synonymous with (although it may well embrace and indeed be manifested by) social participation, social integration, social cohesion, and/or social support individuals can access or be barred from on account of their membership in groups and/or formal and informal institution.

The author concluded that research on social capital focused on two main types of social capital: bonding (between individuals in a group) and bridging (between groups). Each type of social capital has cognitive and/or structural component(s) and may operate at micro and/or macro level(s).

Patulny and Svendsen101 advanced a framework based on the distinction between bridging and bonding capital and qualitative and quantitative methods. Derose and Varda102 reviewed over 2000 abstracts and found that 21 examined some measure of social capital. As others reported previously,100 they found a lack of congruence in how it was measured and interpreted. This made understanding and explaining the relationship with health status problematic. They argued for the separate use of the concepts of bonding, bridging, and linking social capital,103 and for further research into cognitive, behavioural and structural dimensions.

The following are all cross-sectional studies and are therefore subject to some limitations. De Silva et al. 104 used multilevel modelling to examine the impact of individual- and community- level social capital and common mental disorder (CMD) in women in four countries. They found that individual social capital is related to a reduced likelihood of having a CMD but that lower structural social capital increased the likelihood, although not consistently in all locations. A similar finding of an absence of a relationship between structural social capital (organisational membership) and health and well-being has been observed in China. 105

d’Hombres et al. 106 also found that individual levels of trust are positively and significantly related to health, and that social isolation has a negative effect on health. In contrast, organisational membership (contextual) was found to be unrelated to health. Higher subjective levels of trust and feelings of safety have been found to be associated with better mental health. 107

Irwin et al. 108 found that in the most resource-poor populations, social capital added to the explanation of depressive symptomatology beyond that explained by social support alone.

Stafford et al. 92 tested stress buffering and main effects models in data from 9000 respondents, and found no main effect of social capital on CMDs. Attachment to the neighbourhood was associated with higher reporting of CMD, and contact with friends was associated with lower reporting of CMD, but these findings were confined to deprived areas only. An association between social capital, neighbourhood deprivation and hospital admission was observed in a cohort study in Sweden. 103

Others have found that individual measures of social capital, such as trust and reciprocity, are related to poor health status and health-related QoL. 109

Cultural issues in the measurement of social inclusion and social capital

A number of studies have found that social capital and social inclusion results differ in different communities, for example rural and urban communities,22,110 and in different cultures. 104,105,111–113 In an interesting cross-cultural study of social capital using the Short Adapted Social Capital Assessment Tool (SASCAT), De Silva et al. 104 argued that traditional psychometric testing is not sufficient to adequately validate multifaceted social capital tools for use in different cultural settings. The SASCAT has good psychometric properties, but interviews with respondents found that they did not always interpret the questions correctly, or at least in the way intended by the interviewer. We would expect the same issues to arise in the use of social inclusion measures in different cultures. The most culturally diverse of our focus groups in Phase I did not produce material about social inclusion that was radically or even marginally different from the dimensions identified by the other groups. The SASCAT was successfully modified to increase the validity of its use in different cultures, and we would expect the same work to be needed to demonstrate the cross-cultural validity of our measure and other measures of social inclusion.

Measures of social capital

A number of new measures of social capital have been developed since our Phase I review in 2005. 104,112–117 Nevertheless, because our interest is in the measurement of social inclusion we have not explored these in more detail.

Measures of social inclusion

A measure of social inclusion was developed for use in the evaluation of community-based arts for mental health projects in the UK. 93 Three scales were constructed measuring social acceptance, social isolation and social relations; these had good internal consistency, as did the measure as a whole. Correlations with empowerment and a mental health measure were described as ‘reasonable’. The measure is acceptable but test–retest reliability and construct validity have not been demonstrated.

Two small-scale samples of individuals from a psychiatric rehabilitation setting (total number = 54) were interviewed with a view to assessing the reliability of a composite measure of social inclusion. 93 The interview covered sociodemographics, domain-specific socially valued role functioning, social support, community integration and the experience of stigma. Test–retest and internal consistency results were promising, but some items with low test–retest reliability were removed from the stigma and community integration subscales. The sample came from one small psychosocial rehabilitation service and the sample was largely male (80%), which reduces the generalisability of these findings somewhat.

Social participation measures68,118 were covered in the original review and there have been only two new measures published since. 71,95 Both of these measures have been used on limited patient populations and neither has been conclusively tested in terms of psychometrics, acceptability, construct validity or responsiveness.

We therefore find ourselves in a similar position as we were in at the start of the present project, with no established measure of social inclusion available for use in the general population or community mental health service settings.

Concept mapping

The method we used in the group was concept mapping, which is useful when you want to explore the meaning of a concept to participants. Concept maps were developed in the 1970s in the course of Novak’s research programme, at Cornell University,119 where he sought to follow and understand changes in children’s knowledge of science. During the course of this study, researchers interviewed many children, but found it difficult to identify specific changes in the children’s understanding of science concepts by examination of interview transcripts. Out of the necessity to find a better way to represent children’s conceptual understanding emerged the idea of representing children’s knowledge in the form of a concept map.

Concept mapping is a type of structured conceptualisation that can be used by groups to develop a conceptual framework that can guide evaluation or planning. A good way to define the context for a concept map is to construct a focus question, i.e. a question that clearly specifies the problem or issue the concept map should have to resolve. Every concept map responds to a focus question, and a good focus question can lead to a much richer concept map. In the typical case, six steps are involved:

1. preparation (including selection of participants and development of focus for the conceptualisation)

2. the generation of statements

3. the structuring of statements

4. the representation of statements in the form of a concept map (using multidimensional scaling and cluster analysis)

5. the interpretation of maps

6. the utilisation of maps.

Concept mapping encourages the group to stay on task, results relatively quickly in an interpretable conceptual framework, and expresses this framework entirely in the language of the participants. It also yields a graphic or pictorial product, which simultaneously shows all major ideas and their inter-relationships.

In our concept mapping process, each participant was given a batch of yellow Post-it^® notes (3M, St. Paul, MN, USA) and asked to write on three of them what the term ‘social inclusion’ meant to them. When these were completed, someone stuck their Post-it notes on to a flipchart and then others followed suit, placing ones that seemed related closer together. Opportunities were then provided to discuss the ‘map’ and reorder it, if required, through group discussion. The resulting concept maps were all recorded and photographed by the research staff, who subsequently produced nine different ‘maps’. Details of the maps are given in the final report of Phase I. 120 A summary is provided below to explain the major features.

• The first-year university social work students emphasised people’s rights to access facilities and material goods.

• The general public group, while mentioning some positive aspects of inclusion, made the greatest number of negative remarks about excluded groups of people.

• The mental health/social inclusion group emphasised the difference between individual inclusion and social inclusion.

• The women’s group had a dynamic view of inclusion and exclusion in which any one person was constantly moving between the two.

• The mental health academic group stressed the importance of material inclusion, but was also one of the groups that pointed out the rhetorical nature of much of the government portrayal of inclusion.

• The older people recognised that the nature of inclusion changed over time and between generations.

• The mental health service users (MHSUs) in the groups tended to focus on difficulties of accessing mental health services and on their experience of being excluded by the wider society.

Although there were differences in the emphasis placed on aspects of inclusion, there was a great deal of commonality in the domains of inclusion/exclusion, and these were not dissimilar to the domains of generic QoL measures. There was also a broad measure of agreement that social inclusion includes both objective and subjective aspects, and that even if objectively included one can still feel excluded, and if objectively one is excluded – say in the work domain – by not having employment or meaningful activity, one can still feel included within one’s community or family.

Social inclusion is therefore clearly multidimensional and almost certainly dynamic, i.e. subject to change over time. It should therefore be possible to reach agreement on the content of an inclusion measure, with subjective and objective ratings within each domain.

Nevertheless, it was also clear from the results of this exercise that one has to take account of the location of the person within society, given that opportunities to participate and to be included vary by location and social group. It will therefore be necessary to assess the availability of opportunities in the social communities of which the person is a part. For clinical, operational or service use, it is also worth noting the extent to which each individual wants to alter their level of inclusion. It does not follow that all of us, most of the time, want to be more included. 70 There are times when, in most life domains, one wishes to moderate one’s level of inclusion, say, for example, in terms of contact with one’s extended family. Pressure to conform was identified as an issue for several of the groups.

On the basis of the statements generated by the groups we produced an overarching model (Figure 1) using Nvivo version 8 (QSR International, Doncaster, VIC, Australia). The model retains the split between objective and subjective factors, recognises individual differences, and shows how opportunities and choices relate to material domains. These factors, along with personal feelings about inclusion, result in participation and engagement.

FIGURE 1.

The model.

If we unpack the model a little further we see that material factors relate to having work, education and adequate health and housing, along with a group of financial factors (Figure 2).

FIGURE 2.

Material circumstances.

Negative feelings of exclusion (Figure 3) relate to issues around disadvantage, barriers and stigma, which can heighten feelings of loneliness and isolation, but there is also pressure to conform.

FIGURE 3.

Negative feelings of exclusion.

Positive feelings of inclusion (Figure 4) relate to acceptance, being valued, having a sense of purpose, achievement and belonging, while celebrating diversity; this promotes acceptance and inclusion.

FIGURE 4.

Positive feelings of inclusion.

Based on the concept mapping exercises, we developed a matrix of social inclusion that incorporated the domains identified by the groups. This is presented in Table 3. Within each domain, opportunities will be enabled or constrained by availability and the ability to access that which is available. Each individual will have a view about the perceived availability of opportunities for inclusion and will be able to report on the extent to which they are able to avail themselves of these opportunities. They will then have a view about whether their participation or access has been of benefit to them, and finally they will be able to express a view about whether they wish for more or fewer opportunities to participate or to be included. One needs to recognise that the extent to which individuals participate is a personal decision and that for some people choosing to participate less can be the preferred option. An increase in feelings of inclusion can therefore occur even when participation is in decline.

This was our view at the conclusion of Phase I. Phase II involves the production of a measure with which to test these ideas empirically.

Instrument development and pilot testing

The first component of the study focused on the determination of appropriate questions to populate the domains highlighted in Table 3, as identified in the Phase I study, and which corresponded with the domains identified by others. 26,32,45

In order to comply with the objective of facilitating normative comparisons, our search for questions focused on UK national surveys of the general population and other sources of normative data.

All of the surveys listed in the Centre for Applied Social Surveys (CASS) Question Bank121 [known as Surveys Question Bank (SQB)] and UK Data Archive122 were searched for suitable objective and subjective questions to populate each of the domains identified in Phase I. Full details of the databases searched and the questions selected are given in Appendix 2. Where there was more than one candidate question, the research team discussed the merits of each question and made a decision on the basis of the clarity of the wording, accessibility of the question’s source survey data, and whether the source permitted the downloading of individual-level data (which was uncommon) or disaggregation of the results to a local service area level (local authority or ward, which was more common), as this would facilitate general population comparisons within specific localities.

A draft questionnaire, which included fixed-response and free-text questions structured by domain, was prepared by the research team. The questionnaire was cognitively appraised prior to piloting, by research team members and co-applicants (one of whom is a specialist in the field of social exclusion and mental illness), and by experts in survey design based at Opinion Research Services (ORS), the organisation tasked with survey data collection. An example of feedback from this process is provided in Appendix 3. In addition, the research team met with the ORS staff on several occasions to amend the wording and routing of the draft instrument. An example of amendments made at one of the meetings is included in Appendix 4.

In order to check ease of understanding, appropriateness and acceptability, the draft instrument was then pre-tested in a small random sample of the general population in a south Wales city, interviewed by ORS staff (n = 8), and a convenience sample of eight social researchers based in the host institution for this grant. Questions were considered individually and as part of the whole measure. Mechanical issues such as question order, routing and filter questions were tested, and, where necessary, adjustments were made. Ambiguous wording and unclear or misleading questions were identified and adjusted. Questions that were likely to have a low response rate, for example questions on personal finances, were adjusted and prompts added in some instances in order to encourage a response. For example, instead of solely asking ‘What is your personal income before tax?’, the question is followed by a prompt, to enable respondents to locate their income within bands for weekly, monthly or annual amounts. This component of the study was completed over a 6-month period.

Instrument modification and preliminary testing

This component of the study involved modifying the draft instrument on the basis of results generated in the pilot exercise and using this amended version of the Social and Community Opportunities Profile (SCOPE) (referred to subsequently as the SCOPE ‘long’ version) in a community survey, in order to generate data for data reduction and psychometric analyses.

Data analysis

Psychometric evaluation

The psychometric properties of SCOPE (long and short versions) were examined in order to assess the reliability and validity of its items and scales. The reliability of a scale relates to its freedom from systematic error, in that it elicits the same or very similar responses irrespective of who asks the questions (inter-rater reliability), is stable over short periods of time (test–retest reliability) and the items included within it measure the same underlying attribute (internal consistency). The validity of the scale relates to the extent to which it measures what it is supposed to measure in terms of its content (content validity), and how it relates to similar measures of the same construct or similar constructs (construct validity), often within different population groups to demonstrate similarities (convergent validity) or differences (discriminant validity).

Summary

We found objective questions on key variables by searching existing survey databases [mainly Office for National Statistics (ONS) ones]. We used the delighted–terrible rating scale from previous work for all the subjectively rated items. All suitable candidate questions were retained for the ‘long’ version, resulting in a 121-variable version. This was the version we then subjected to preliminary testing in a community survey (see Component 2: preliminary testing).

Summary

We conducted a community survey in five locations in England and Wales, resulting in 252 completed versions of the ‘long’ SCOPE. The sample was older than Census data, but not significantly so. There were employment status differences and related differences (car ownership, for instance) indicating that our sample was significantly disadvantaged compared with the general population. Consequently, it was no surprise to find higher rates of mental ill health in our sample. In order to be certain that the mental health score threshold resulted in a sample with a very likely CMD we set the cut-off point so that 16% of the sample had CMD, the same as in the NPMS. This resulted in a sample with very similar demographics to a sample of MHSUs in contact with community mental health teams (CMHTs) collected to examine sensitivity and responsiveness (see Table 8). We used the healthy community sample, the CMD cases and the MHSU group to conduct discriminant validity analysis. We consider this and other psychometric features of the long version in the next section.

Internal consistency (long version)

As highlighted in Table 4, the SCOPE long version includes objective items relating to opportunity and participation and subjective assessments of well-being, opportunities and Perceived Opps in eight domains. An additional subjective item relating to overall inclusion is also included. These items are shown in Table 9.

The mean of domain ratings of SWB, SatOpps and Perceived Opps were calculated to form three cross-domain ratings. The SWB and SatOpps indices each have a minimum score of 1 and a maximum score of 7, and the Perceived Opps index has a minimum score of 1 and a maximum of 5. This approach of producing an average score rather than a total score (using the SUM function) is advantageous in that missing values have less impact, but the smaller range might affect the capacity to discriminate between groups and over time.

The desired change to opportunities index was produced by recoding the open-ended desired change questions into bivariate items where ‘yes’ = 1 and ‘no’ = 0, and summing these scores to give a minimum possible score of 0 and the maximum possible score of 10.

Twelve of the fourteen objective items relating to access needed recoding into binary scores on which a score of 1 was positive, indicating greater inclusion; two items relating to the number of friends and number of close friends were recoded and categorised, for example, as zero friends = 0, some (i.e. 1–10 friends) = 1, and many (i.e. > 10 friends) = 2. These recoded variables were then summed to create a rating of the extent of inclusion [Fourteen-Item Objective Opportunity Rating (O14)], on which the minimum possible score was 0 and the maximum possible score was 18. Similarly, 12 of the 13 participation items were recoded into binary variables on which a score of ‘1’ indicated a positive situation; use of sports and entertainment facilities was reclassified as ≤ 3, 4–5 and 6+ and scored 1, 2 and 3. These variables were also summed to create a participation rating [Thirteen-Item Objective Participation Rating (P13)] on which possible scores ranged between 1 and 18.

The internal consistency of the cross-domain indices was tested using Cronbach’s alpha coefficients, and the results are shown in Table 10.

Alpha coefficients of between 0.7 and 0.9 indicate good internal consistency for SWB, Perceived Opps and SatOpps scales in the sample as a whole, and when analysed by mental health sample groups.

The changes to opportunities items have a low alpha coefficient, suggesting a lack of internal consistency between items, which raises questions about the construct being measured and whether or not the items constitute a reliable scale. Nevertheless, low alpha values such as this are not uncommon in short scales (i.e. with < 10 items) and in these instances it is more appropriate to report the mean inter-item correlation,135 where a range between 0.2 and 0.4 is regarded as optimum;136 the changes to opportunities index did not reach these optimum levels, confirming that its items should not be used as a scale. Nevertheless, the changes to opportunities items have a higher alpha coefficient and good inter-item correlations in the MHSU sample. These results suggest that the changes in opportunities items should be excluded from the short SCOPE, but might be included alongside the short SCOPE items in studies of MHSUs in clinical settings.

Low alpha coefficients and poor inter-item correlations for O14 and P13 indicate that neither constitutes a reliable scale. The items contributing to these ratings are included later in the chapter (see Tables 15 and 17). Because the research team was tasked with developing a social inclusion index that captures objective as well as subjective items and can locate feelings about inclusion within the context of the availability of, and participation in, specific activities, these items are retained in the short SCOPE despite their lack of internal consistency. Nevertheless, the team recommends that O14 and P13 items be considered and analysed individually and not as scales.

The SWB scale, the SatOpps scale and the overall inclusion scale were all significantly negatively skewed. Skewness was tested using the following formula: skewness/standard error of skewness. Values falling outside the range of ± 1.96 are skewed. For the analyses that follow, the skewed variables were transformed to meet the assumption of normality required for parametric tests. Unless otherwise stated, the significance, magnitude and direction of effects did not differ in analyses of untransformed and transformed data. Consequently, where this is the case results based on untransformed data are presented for ease of interpretation. The skewed variables were transformed using reflect and logarithm [new variable = log10 (K – old variable), where K is the largest possible value of the variable + 1]. Tests for significant skew were repeated following the transformation and all three variables fell within the normal distribution range. The Perceived Opps scale had a normal distribution originally, and did not require transformation.

Analysis of transformed and untransformed data produced the same results in terms of the nature, magnitude and significance of differences, so the results that follow are based on the original untransformed data, for ease of interpretation.

Discriminant validity (long version)

In order to determine whether or not SCOPE subjective scales were capable of discriminating between known groups, the impact of mental health status was examined using one-way between-groups ANOVA, with Tukey HSD post hoc adjustments. To explore the discriminant validity of the scales further, the analysis was repeated using the age groups ≤ 64 years, 65–74 years, 75–84 years and ≥ 85 years. Independent samples t-tests were conducted to compare the subjective SCOPE scale scores for those self-reporting a LLTI and those reporting no illness (no LLTI).

Construct validity

As Phase I of the study had not identified any existing measures of social inclusion, construct validity could not be assessed against a gold-standard measure, and instead was assessed against related measures of participation and social capital. We expected that a proportion of the concept of social inclusion would be made up of social participation, but we were unsure whether or not they might actually be the same thing. We anticipated, therefore, that the Berry community participation measure69 would be related to the participation items within the SCOPE. We also expected that social capital and inclusion would be related, as to create and make use of social capital one has to interact with others, and so it too would be related to participation. In order to examine these inter-relationships we compared the three social inclusion measures of Perceived Opps for inclusion, SatOpps, and overall SWB with the single question about how socially included people felt, and with the objective opportunity and participation items, the Berry measure69 and the social capital measure (RG-UK). 114 We expected that these would all be closely related, but were unsure how distinct they might be.

Table 19 shows that the three SCOPE scales correlate more highly with each other than with the measures of participation and social capital. As one might expect, Perceived Opps for inclusion and SatOpps are most highly correlated and share almost two-thirds of their variance. SWB is also highly correlated with Perceived Opps for inclusion and SatOpps for inclusion, all of which are closely associated with the overall inclusion rating; the shared variance with the overall rating is highest among items measured on the same seven-point scale, namely SatOpps and SWB. Objective indicators of opportunity and participation are also correlated significantly with the subjective scales, although the magnitude of the associations was somewhat smaller.

These results are consistent with our previous understanding of subjective measures, which can be expected to be highly correlated with each other and less so with objective indicators. 159 The results indicate that the SCOPE items clearly tap in to the same construct, but support our view that the construct is multifaceted and includes several different components.

Participation (as measured by Berry) was only moderately correlated with the three SCOPE scales and the overall inclusion rating, sharing between 13% and 17% of the variance. Nevertheless, the SCOPE-P13 items correlated more highly with the Berry measure, sharing just over one-quarter of its variance. One would expect that participation would have a reasonably strong relationship with elements of our social inclusion measure, reflecting the fact that in order to be and to feel included one has to participate in society in some shape or form. Clearly, the SCOPE-P13 items are more related to the objective aspects of participation included in the Berry measure than are our subjective indicators. It is reassuring, however, to find that social inclusion is not the same thing, conceptually, as participation alone.

Social capital was associated more closely with SCOPE-O14 items and the Perceived Opps for inclusion scale than any of the other SCOPE scales, but still shared less than one-quarter of the variance, suggesting that the two concepts are related but not the same.

Finally, social capital is related to but not the same thing as participation per se, and shares only 18% of its variance.

Acceptability

In addition to completing the SCOPE long version, the MHSU group were also asked to complete an evaluation form to assess acceptability, completion times, etc. The views expressed in the evaluation were taken into account when creating a shortened version of SCOPE. The mean completion time among this group was 37 minutes (SD 16 minutes; minimum 15 minutes, maximum 2 hours). Most people felt that this was too long, with 85% finding it either much too long or too long (n = 37), which is unsurprising given that the purpose of collecting these data was to facilitate data reduction. Participants were asked to rate how important the life domains in SCOPE were to their lives. Overall, 76.9% felt that the domains were relevant compared with 23.1% who did not (n = 39). Table 20 shows the rating for each life domain.

Forty-five per cent of respondents felt that one or more questions were inappropriate. Of the 12 people who outlined which questions were inappropriate and why, eight did not want to answer questions about money, two found the questions too personal and two felt that the questions did not capture their own experiences.

Only 18.9% felt that some questions had not made sense; three people made further comments and highlighted repetition as a problem, which was partly a consequence of including the three different measures of social inclusion, participation and social capital, in the long version.

Sixteen per cent felt that certain additional questions should have been included; further comments elicited single-person suggestions for asking about pets, home interests, involvement with the police through illness, more relevance to older people, sexual health and emotional well-being.

Almost one-half of respondents felt that there were some questions that they would rather not have answered. Thirteen people commented further, with nine saying that they did not want to answer questions about finances. The other four all said that some questions were too personal and intrusive.

Summary

In a sample of people with mental health problems the acceptability of the SCOPE long version was high (with the exception that they said it was too long). The change to opportunities index was not a scale and was excluded from the short version, but had acceptable properties in the MHSU group so might be added to surveys for this group only. Objective and participation variables also had low alpha coefficients and should be analysed individually and not as a scale. Discriminant validity was demonstrated for the Perceived Opps scale, the SatOpps scale and the SWB scale. We recommend, however, that the last is left out of the short version of SCOPE, on the grounds that this is measured in QoL assessments such as the MANSA.

A substudy showed that there are conceptual distinctions between ‘participation’ and social inclusion (shared variance between 13% and 17%), social capital and social inclusion scales (< 25% shared variance), and social capital and participation (shared variance 18%).

Data reduction

Items that had > 10% missing data or little or no variance, that overlapped considerably (r > 0.7) with other item(s) or had low factor loadings across all domains were excluded on the basis they added little to the measure or appeared not to be important components of social inclusion as conceptualised here. As a large proportion of respondents did not like the financial questions, we reduced the number of financial questions and amended some of the response options.

Using descriptive statistics we identified five questions that could be omitted from the questionnaire on the basis of lacking variance, i.e. > 90% of respondents endorsed one response. These items related to internet use, housing tenure, adult learning and contact with children aged < 18 years not living with the respondent. Nine items were omitted because they shared > 50% variance (r > 0.7) with another item in the questionnaire. These also related to internet use and contact with adult children living apart from the respondent, as well as work and current study for qualifications. Decisions about which item should be omitted favoured individual-level questions over household-level questions, and engagement in activities over place of engagement. Six separate items relating to face-to-face and telephone contact with parents, children and other family members were replaced by three items focusing on any contact with each of these family groups. Two subjective items relating to feelings about the amount of family contact and feelings about family relationships were excluded in favour of feelings about the range of opportunities for family contact.

The remaining items (minus four demography questions) were then entered into domain-level principal components factor analyses to identify the core components of participation and engagements in each domain, and to determine whether inclusion at this level is unidimensional or multidimensional. The item loadings on the component matrices for each life domain were examined and the highest loading factor items were retained. Although we had intended to use a factor loading cut-off of 0.6, this was raised to 0.7 in most instances in order to reduce the number of items being retained, and so reduce the burden on respondents in future use. The exploratory factor analyses are summarised in Table 21 and reported in more detail in Appendix 8.

For most domains the underlying structure was related to objective and subjective indicators only. However, for leisure and participation, finance, family and social life, and work (for people not in work), desired changes were also an important factor.

The 24 items included in the leisure and participation model related to seven underlying factors. These focused on the availability of and SatOpps for participation, use of, total involvement in and desired changes in participation and leisure, citizenship, and satisfaction with one’s own leisure activities.

The 10 housing and accommodation related items had an underlying structure based on socioeconomic indicators, attachment to area and satisfaction with the opportunities to access suitable housing. Safety was associated with two main factors: experience of crime and satisfaction with safety. Two main factors emerged in relation to work, which were based on SatOpps to find work among people who were employed and unemployed; for those in work, the actual number of hours worked per week was important, whereas having a desire to change employment status was the important thing for people not in work.

The 12 finance items loaded on to five factors, two objective relating to debt and social capital (having someone from whom they could borrow money if necessary), desire to change income, and subjective indicators of opportunities to increase income and satisfaction with income.

The underlying components of education and health also focused exclusively on objective and subjective indicators, the first relating to highest level of qualification, participation in learning opportunities and satisfaction with those opportunities. Health was associated with objective mental health status, service contact in relation to physical and mental health, SatOpps to access health-care services and satisfaction with one’s own health.

Finally, family and social engagement was associated with six factors relating to socialising, family contact, SatOpps for friendship and family contact, and desired change in these areas. These results are reported in more detail in Appendix 8.

These analyses produced a measure of 48 items (including four demographic descriptors) (see Appendix 11).

The decision to retain and reject factors was tested using Horn’s parallel analysis129 – see Appendix 12. This analysis did actually suggest that additional items could be excluded, resulting in a 21-item questionnaire (Mini-SCOPE) that had poor internal consistency (α = 0.47) and in our view could not be used usefully in clinical practice. These reliability and pragmatic issues informed our decision that the additional items should be retained in the SCOPE measure. The Mini-SCOPE is included in Appendix 13, for those who are interested.

In accordance with the protocol, we also conducted within-domain item response analyses undertaken using MSP, which did not identify any items that had not been retained by factor analyses. The MSP results (see Appendix 14) confirmed that, within each domain, satisfaction items, opportunity items and objective items fail to produce a single scale, which is consistent with existing knowledge based on conventional analysis and our understanding of the nature of the within-domain items. Unlike the conventional analysis, MSP analysis failed to provide cross-domain scales. Within-domain analysis confirmed that SatOpps and Perceived Opps items are related and can be used reliably as two separate scales. MSP analysis suggested that some objective items could be combined to produce within-domain scales, but as the number of items included was usually very small (n = 2), this appeared to have little merit over using the individual items themselves. We therefore retained the separate satisfaction, opportunities and objective items, and retained the questions that both forms of analysis identified as scale components.

Table 22 shows the items that were selected by both the factor analysis and the item response analysis, all of which are included in the short version of SCOPE.

Although most domain-specific SWB ratings loaded on factors along with other subjective ratings of access and opportunities, in the interests of brevity and on the basis of their overlap with Perceived Opps and SatOpps scales, SWB items can be excluded from the SCOPE without detriment to its measurement properties. This decision was made in the knowledge that subjective items are included in QoL life measures such as the MANSA, and on the basis of our belief that social inclusion is a component part of QoL rather than QoL being a part of social inclusion. In the safety domain, two items relating to general and personal safety were replaced by a subjective rating of opportunities to live safely in the area.

Similarly, some items that loaded highly onto the same factor were very similar to and correlated with each other. In this instance, the research team decided, jointly, which item should be retained on the basis of acceptability of the question to respondents, factor loadings and the effect on internal consistency of scales, of removing specific items. Subjective items relating to opportunities to improve health and family and social life were reworded for purposes of clarity.

Finally, and conversely, a small number of objective items relating to being a victim of crime, employment status, personal income, etc., did not reach the amended factor loading cut-off point of 0.7 but were usually in the range of 0.6–0.7, which was the cut-off point we had intended to use originally, so these items were retained for contextual purposes.

On this basis, we were able to reduce the SCOPE from 121 to 48 items (including the four demographic profiling indicators).

Summary

All of the SCOPE domains had characteristics that made them suitable for factor analysis (correlations of ≥ 0.3 present in correlation matrix, KMO > 0.6 and statistically significant Bartlett’s test of sphericity). Items that had > 10% missing data or little or no variance, which overlapped considerably (r > 0.7) with other item(s) or had low factor loadings across all domains were excluded on the basis they added little to the measure or appeared not to be important components of social inclusion as conceptualised here. As a large proportion of respondents did not like the financial questions, we reduced the number of financial questions and amended some of the response options. We removed the SWB ratings, as these are adequately covered by measures such as the MANSA. These procedures reduced the SCOPE long version of 121 items to the SCOPE short version of 48 items. We next examine the psychometric characteristics of the SCOPE short version.

Reliability and validity

Alternative approaches

Summary

The short version includes items that almost all students thought were very or quite important to overall social inclusion. We found that there was little difference from the results of the construct validity testing of the SCOPE long version, and that there was a higher correlation between the SCOPE scales than with the measures of participation and social capital; the highest correlation was between Perceived Opps for inclusion and SatOpps for inclusion. The objective indicators correlated significantly but less strongly with the subjective scales. Discriminant validity resulting from Horn and MSP analysis was poorer than the conventional factor analytic solution. Neither the Horn nor MSP analysis produced convincing evidence for the existence of any useful scales.

Service 1

One community mental health service based in London has the specific aim of improving social inclusion in people with severe mental health problems. Agreement was reached with this service for participation in the study and on the methods by which data would be collected; disappointingly, in the event only one completed form was returned.

The SCOPE was piloted in this team during September and October 2009. It was primarily used in the setting of a gym group, as the team itself was not taking on new referrals for other social inclusion activities at that time. As many of its domains were not relevant to the activity of the group, only the health and leisure domains were completed (n = 5). One full SCOPE was completed (taking 1.5 hours) and three practitioner feedback forms were completed on their experience of using SCOPE.

The following observations were noted in a feedback meeting with the team on 20 October 2009. The team members primarily commented on the clinical application of SCOPE in their setting, which they acknowledge is quite specialised and atypical for mental health services. Their work is goal orientated and they did not think that the SCOPE could help with goal setting (not that the SCOPE was designed to do so). They suggested that the SCOPE may be more appropriately piloted in a CMHT setting. One problem relating to the introduction and use of the SCOPE was that they also routinely use other outcome measures such as More Effective Community Care (MECCA) [now known as DIALOG (scale for more effective community care of psychosis)]. While this would have provided an opportunity for further construct validation, team members felt that using the SCOPE in addition to existing measures was burdensome and did not provide added value.

Some team members felt that SCOPE was not person centred and the census-type questions were not amenable for use in a clinical setting. The closed questions did not allow for elaboration of the service user’s views. Some found the ‘delighted–terrible’ scale difficult to complete, although this is not a problem reported in previous research and evaluation, despite its extensive use in mental health settings internationally. Finally, they felt that it had limited clinical applications in their team, but may be more beneficial for care coordinators.

On a positive note, they felt that the domains were largely appropriate for a social inclusion measure and that the questions about the social environment may be generally useful for care planning. The community focus is useful for a social inclusion service so if used systematically it may produce useful data for the service. They also felt that it could be self-completed by more literate service users.

In summarising this feedback, one of the applicants (MW) concluded that the largely negative response to SCOPE from this team reflected more on the way the team operated than on any shortcomings that the SCOPE might have. In the sections that follow it becomes clear that this conclusion might bear some truth as, in contrast, most of the feedback from other services in London and Wales was positive. Importantly, most service users did not take long to complete the short version of SCOPE.

Service 2

As a result of the response (or lack of response) from the first service, it was necessary to revise our plans and engage with other services. We reached an agreement with an assertive outreach service, which also aims to tackle social exclusion, that is located nearer to the research team’s base, in Newport, Gwent. Because time was short, we organised two methods of assessing responsiveness, one retrospective and one prospective. The team in Gwent has completed a retrospective SCOPE (based on case records, standardised assessments and continuous worker contact) on service users known to the team since its inception, by workers who have known them since the time of referral. These data were supplemented by current SCOPE data collected in the form of an interview with the same users. Retrospective assessment is subject to potential bias of recall, and also of inter-rater reliability given that workers tend to rate things differently to the service user themselves. 160 Therefore, we have also arranged for a follow-up assessment 12 months after the first interview with the service user, which will constitute the final prospective SCOPE field testing of responsiveness in clinical practice where an intervention has been provided.

In the retrospective method, two support workers who had known the patients well since they came to the service completed the SCOPE using case records and personal opinion to rate the patients as they were when they were accepted into the service. They then interviewed the patients again and completed a current SCOPE. Paired t-tests, correlation and repeated measures regression analysis have been undertaken on these data. For the same service users a repeat interview after 12 months was arranged. Only a small number of cases are available in the prospective group within the study time period, but these data will go on being collected by the service beyond the end of the funding period.

A much more positive feedback came from the Assertive Outreach team manager (AG), who said that the SCOPE was clear and easy to complete and the time taken was acceptable. The two support staff who undertook the retrospective ratings found it straightforward and acceptable.

Service 3

The third service is an independent sector service and has hubs across south Wales. The project that we engaged with aims to tackle social exclusion in a mixed client group, including people with mental health problems and those who are homeless. People access the service through referral from community mental health services. The service offers part-time courses, a social club, and life skills development. We offered the users of this service a £10 high street shopping voucher per questionnaire as an incentive to participate. Forty people have completed a Time 1 questionnaire. Given the nature of the service provided in this setting, user attendance at project meetings is unpredictable and only 11 have completed the second SCOPE after a 3-month period.

Data collection is ongoing in this service but, again, the final follow-up data will fall outside the project timescale.

Given the small follow-up sample size the analysis that follows is presented primarily to show how we propose to assess change, rather than to give a generalisable set of findings. Guidance on scoring and assessing change is included in the SCOPE User Guide (see Appendix 7). Although this sample may not be an ideal one in the sense of demonstrating change (as Time 1 is not a genuine baseline measure at the start of the intervention) if inclusion improves incrementally over time then this ought to be observable in such a sample.

Sensitivity to change and responsiveness

Our approach to these issues is described in more detail in Appendix 5.

Sensitivity to change is ability to capture any change at all, and responsiveness is the ability to capture clinically important change. Simple changes on the SCOPE, such as not having a job at Time 1 but having one at Time 2, are the equivalent of clinically significant change, in that they show a major feature of social inclusion has been achieved. Losing a job would be equally significant in terms of weakening inclusion.

For continuous variables and scales, Cohen’s effect size seems to the most appropriate to use; it is the ratio of the mean change to the SD of the baseline scores.

Paired-sample t-tests were conducted on the SatOpps score, the Perceived Opps score and the overall inclusion score in order to examine sensitivity and responsiveness over time. We then explored further by conducting paired-sample t-tests on the components of the SatOpps scale and the Perceived Opps scale.

Service 4

Service 4 is a residential rehabilitation unit that aims to provide short-term placement (maximum of 2 years) before users are able move on to more independent living arrangements. The service targets adults who are all on enhanced Care Programme Approaches (CPAs). Research staff met with HK (Senior Lecturer in Rehabilitation Psychiatry) and RB (Team Manager) on 25 February 2010 to establish the prospective responsiveness test. Team members were not able to devote staff time to administering questionnaires but were willing for research staff to administer them. Research governance approval was sought on this basis, and granted. Feedback on the SCOPE was positive, and a small number of minor issues were resolved (see Appendix 17). After this visit the team agreed to become involved with a more local (London) medical school social inclusion research project, and postponed the meeting at which data collection was to be agreed and conducted with service users. No further meetings took place after this.

Service 5

Service 5 is a user-led independent sector service, based in Cardiff. This service takes a holistic approach to overcoming depression through guided self-help, building the foundations for sustainable and long-term well-being. The service also provides information, practical resources, services and training to promote the development of recovery skills and strategies.

Mental health service user group

Concerns that the nature of the MHSU sample might inhibit capacity to demonstrate discriminant validity were unfounded as the many inclusion indicators differed significantly between the recovering MHSU group and the MHC group. MHSU2 scores were similar to those for the MHSU group. The fact that the SCOPE has demonstrable discriminant validity in these samples suggests that this particular psychometric property is underestimated here, and that the SCOPE would be even more capable of differentiating between MHL, CMD and MHSU populations in evaluations or trials in which group status is determined by diagnosis or by illness severity.

The mean MHSU rating for SatOpps was significantly lower than those for the MHC group. Perceived Opps did not differ between these two groups, but this finding is plausible given that the MHSU group were in recovery, as perceptions about opportunity might be affected more than other indicators of inclusion by current depressive symptoms. The MHSU group were as likely as, or in some cases more likely than, the MHC sample to be actively participating in activities. For example, the MHSU group were considerably more likely than the others to be involved in a local group or organisation, and to have been in full- or part-time education or adult learning, but they were less likely to be volunteering. These results might be explained by selection bias, in that the group who provided the data are longstanding service users who are relatively stable and who have good social contacts with other service users. Nevertheless, significantly fewer in the MHSU group were employed than in the general population sample. Exclusion from the labour market perhaps enhanced their opportunities for inclusion in alternative groups or activities.

This raises the question as to whether or not inclusion in certain activities is preferable to others, and whether or not objective items should be weighted accordingly to reflect desirability. Some might argue that any weighting should be adjusted for individual choices and needs as in measures such as in the Comprehensive Quality of Life Scale (ComQol)172,173 and the Schedule for the Evaluation of Quality of Life (SEIQOL). 174 Nevertheless, we and others are not persuaded of the need for individual weighting, especially in measures such as the SCOPE, whose domain and item content is determined because of their inherent importance. Serious methodological and mathematical concerns have been reported about using weighting at all,175 and about the use of scales commonly used to rate importance to inform the weighting process. 176–178 These doubts, alongside clinical and practical doubts, provide a convincing argument that specific domains of known importance should always be measured and might lead one to conclude that ‘weighted’ measures have little or no advantage (beyond intuitive appeal) over ‘unweighted’ measures, and that there are good reasons for not ‘weighting’ ratings. 175 Cummins179 was so persuaded by these arguments that he now recommends using ‘unweighted’ ComQol domain satisfaction indicators.

Further work needs to be undertaken on the sensitivity to change and responsiveness in more appropriate clinical samples.

Mentally healthy community sample

The survey methodology used to provide a general population sample including the MHC and CMD population subgroups resulted in a sample that was slightly older, more often retired, and with more LLTI than for the population as a whole.

The question on LLTI, new in the 1991 Census,180 asks ‘Do you have any long-term illness, health problem or disability which limits your activities or the work you can do?’ Dale181 has noted that pre-Census test data on the LLTI correlated well with other data on general practitioner (GP) consultancies and inpatient and outpatient visits to hospital, making it a robust proxy indicator of health status and health service usage; she argues (as do others182) that it provides the only nationally consistent indication of health service needs.

In the 2001 Census,144 one in six people living in a private household in the UK (10.3 million) reported having a LLTI. Rates of LLTI increased steadily with age, as one might expect, for both males and females. Rates of < 10% were observed for people aged < 30 years, but were more than double that figure for those aged 45–59 years. Rates virtually doubled again in the 60- to 74-year age group, reaching 41% for men and 38% for women. Rates of LLTI were similar (within one percentage point) for males and females in each of the age groups up to 59 years. Nevertheless, gender differences were more apparent among people aged ≥ 60 years. In the 60- to 74-year age group men had a higher prevalence of LLTI than women, but the situation was reversed for those aged ≥ 75 years, with more women than men reporting a LLTI.

Our assumption that higher rates of LLTI in our sample were associated with the higher proportion of older people in the sample was not fully supported, however. Rates of LLTI in the younger age groups were considerably higher in our sample than in the Census data, reaching 17.5% in the group aged < 30 years and 29.5% in the 45- to 59-year age group. LLTI rates for those aged between 60 and 74 years were similar but slightly lower, at 34%, than in the Census data (30%). The male–female ratio is similar in those aged < 60 years, but in our sample we found more men than women with LLTI in both the 60 to 74 years and ≥ 75 years age groups. The oversampling of people with LLTI was likely to be a result of our sampling method.

Nevertheless, the impact of this apparent sampling bias was limited, as the overall subjective inclusion rating did not differ between the LLTI and no-LLTI groups (although it approached significance), and both groups were equally satisfied with their current opportunities for inclusion. As might be expected though, the two groups did differ in respect of their perception of the availability of opportunities, with mean scores for the LLTI group being lower than those for the no-LLTI group. SWB was also lower in the LLTI group. LLTI significantly affected use of sports and leisure activities, walking around after dark, and use of full- or part-time education or adult learning. It also adversely affected internet access, employment status and income from paid employment/employment pension. As with the CMD group, the findings may be influenced by the presence of depressive symptomatology, which has not been controlled for in analysis so far.

The results for the economic activity of people with and without LLTI are similar to those reported by the Scottish Government183 – 83% of adults of working age without LLTI were economically active compared with 30.9% of those with LLTI. The Scottish study183 also found, as we did, that a significantly higher proportion of those with LLTI were receiving state welfare benefits.

As one might expect, the proportion reporting physical and mental health problems was significantly higher among people with LLTI. Research undertaken by the PIs in North Wales suggests that people experiencing mental ill health often attribute limitations in terms of experiences of inclusion to their mental health problems, which can make it difficult for people to go out, to socialise and to be confident in company. Therefore, for many people with a LLTI participation and inclusion may be limited as much by self-exclusion as by being excluded by others or by lack of opportunity, but is equally real nevertheless.

One undesirable consequence of random community sampling is that minority groups are not included in large enough numbers to explore their experiences separately; therefore, further research will be needed to explore the inclusion status of minority groups.

Common mental disorder group

The chances that discriminant validity between the CMD group and the MHC group might have been compromised by our choice of threshold and the resultant possible exclusion from the CMD group (and therefore inclusion in the MHC group) of some people showing fewer symptoms of mental ill health also need to be explored. Nevertheless, this investigation needs to be considered in the context of recent research in mixed urban and rural settings, which produced findings that are broadly in line with the threshold for CMD that we chose to use in the survey data. 184 In addition, readers should recognise that repeating the analysis using the recommended threshold of the MHI-5 did not alter the results. 156

Riva et al. 184 found that rates of CMD varied significantly for village populations, but not for those in semirural areas. The average rate of CMD in this study was smaller for villages (14.6%) than for urban settings in ‘other cities’ (18.4%), straddling our chosen threshold of 16% for our sample, which was drawn from rural, semirural and urban settings. The rate of CMD in the most recent NPMS185 was almost identical to the 2000 rate.

Further endorsement that our selected threshold for CMD status was appropriate is provided by a study conducted in one of the south Wales areas included in the present study that also used the MHI-5 (n = 14,669). 156 A comparison of the two studies indicated that our small sample was not atypical in demographic terms or in mental health status. Kelly et al. 156 reported a mean baseline MHI-5 score of 71.1 compared with 75.7 (SD 3.07) in this study (χ² = 0.41, df 1, p = 0.52). The mean age in their study was 56 years and ours was 53 years, and 56% of both samples were female.

The lower levels of participation in the CMD group might be explained by the fact that they were ‘ill’ at the time of data collection, given that this group was identified on the basis of reporting of present symptoms. Consequently, the CMD group probably are participating less than they might do when they are mentally well. In comparison, the MHSU diagnosed group are relatively speaking ‘well’ at the point of data collection (discharged and surviving in the community – but nevertheless the group least satisfied with their mental health). It could also be that some of the common disorders involve phobias, such as agoraphobia or social phobia, which, as with other types of anxiety and some forms of depression, are likely to result in reduced contacts and participation – sometimes due to self-exclusion.

Contribution of authors

Professor Peter Huxley, PI and lead investigator, was involved in the conception, design, execution and writing up of the interim and final reports.

Dr Sherrill Evans, co-applicant and co-investigator, was involved in the conception, design, execution and writing up of the interim and final reports. She was also responsible for overseeing the production of the instrument, supervising research staff and advising on statistical analysis, with the support of experts cited in the acknowledgements.

Ms Sally Madge was the senior research assistant who worked closely with ORS and participating agencies on data collection, and undertook data entry and analysis in spss. She has contributed to the interpretation of data and to the writing of interim and final reports.

Dr Martin Webber was a co-applicant and contributor to the bids in the Phase I and II projects, as well as providing background materials for the literature reviews, commenting on early versions of the instrument and reading, editing and signing off the final report. He also contributed the material on item response theory, including data analysis and writing up, as well as providing advice and materials on the use of the Resource Generator tool.

Dr Tania Burchardt was also co-applicant and contributor to the bids in the Phase I and II projects, as well as providing background materials for the literature reviews, commenting on early versions of the instrument and reading, editing and signing off the final report.

Dr David McDaid was also co-applicant and contributor to the bids in the Phase I and II projects, as well as providing background materials for the literature reviews, commenting on early versions of the instrument and reading, editing and signing off the final report.

Professor Martin Knapp was also co-applicant and contributor to the bids in the Phase I and II projects, as well as providing background materials for the literature reviews, commenting on early versions of the instrument and reading, editing and signing off the final report.

National Development Team (for inclusion) review of measure of social inclusion

Bates190 also makes the distinction between the individual sense of inclusion and the government’s definition of it. He developed a series of key indicators (partly based on work by the Inclusion Research Network with subsequent additions) and these are:

• access to specific services empowerment, participation, employed within

• standard of living income, housing, employment, education, health care

• relationships social networks, social capital, respect and positive attitudes of others.

The questions in these areas are also mostly copied from established surveys, as in the SCOPE measure. Only 11 questions are used, compared with 12 pages in SCOPE. As Bates points out (as did several of the concept mapping groups), the answers to these questions do not give a sense of their personal meaning to the respondent, and ignore the fact that any individual may choose not to interact with family members, for example. He also reminds us that ethnicity and gender are powerful determinants of the experience of exclusion and inclusion, and from the concept mapping results in the present study we should add age to that list. He provides a useful summary of existing instruments and the domains that they cover under the headings demographics, social roles and relationships/psychological. Each of the measures has been assessed using a standard review form, but it is not clear how the list of measures was drawn up in the first place.

Conclusion Bates has undertaken extensive and valuable work on the extent to which services meet inclusion criteria but this is beyond the scope of the present study. Nevertheless, it does mean that there is a tool available for making assessments of service provision against these criteria.

Mind South West (Williamson and Allen 2006)191

From Mind South West; registered charity; www.mind.org.uk comes an assessment based on ‘the human givens’. These are safe and secure; give and receive attention; independence and control; emotionally connected to others; part of a wider community; friendship and intimacy; sense of status within social groups; sense of competence and achievement; meaning and purpose; physical health; spirituality; creativity; and finance. It is a measure for use in mental health services and defines mental health as ‘the capacity to think, feel and act in ways that lead to fulfilling relationships and a sense of well being’, making it an exclusively subjective measure. Under each heading the respondent completes boxes with statements in respect of (e.g. safety) things that make me feel safe; all you feel you have on a good day; what happens on a bad day; and how you would like to find ways of meeting this need. The respondent then gives a score out of 10 to represent how well he or she thinks they meet this need.

Conclusion The tool will have all the strengths and weaknesses of a goal attainment schedule, i.e. while it is highly personalised, it makes direct and meaningful comparison between individuals or group aggregation (except for the score out of 10) more or less impossible.

Developing valued lifestyles support needs pack (Davis and Lindley 1999)192

This measurement tool is also designed for use in mental health services to determine the type and level of support any individual might require. The domains are living in the neighbourhood; getting involved in the community; making own decisions; being respected; building on strengths; money matters; health and fitness; peace of mind; educational aspirations; and work inclusion. Under each heading are about 20 statements (e.g. find places with local social life) rated on a seven-point scale from ‘no help’ to ‘a great deal of help’, and a free form section to put three major choices of help topic. This is to help staff help the person meet these goals, and so is not unlike the Mind South West measure in this respect. All of the domains emerged in the present concept mapping exercise.

Conclusion It is not clear how Davis and Lindley arrived at the domains or the statements. However obtained, the concept mapping exercise provide some post hoc validation for the choice of domain, or vice versa! It is primarily of potential value in service settings to monitor progress regarding inclusion and normalisation perhaps.

Anglia Ruskin/University of Central Lancashire (Hacking et al. 2008)193

This instrument was based on a review, beginning with the Social Exclusion Unit report, adding published and unpublished literature, including service user contributions from websites of the major mental health charities. Instruments were also sought from arts and mental health projects in England, and questions sought from labour force and household surveys. Concepts arising in these materials were mapped against previously used questions by one member of the team followed by team discussion to include, exclude or adapt the questions. Where no questions addressing concepts were found, new ones were created through discussion with the whole team, which includes two former service users. The resulting 22 questions were grouped into the following categories: building social capital (six items); social acceptance (five items); neighbourhood cohesion (two items); stability of housing tenure (two items); engagement in leisure and cultural activities (three items); and citizenship (four items). Initially, they were Likert rated but, following piloting with 15 participants from arts and mental health projects, this was changed to a four-point scale (‘not at all’ to ‘yes’, ‘definitely’). Ninety participants from 22 schemes completed the scale (88 answered over 90% of the questions and the other two fewer than one-third and so were removed). Data reduction techniques resulted in a 19-item measure, covering social isolation, social relations and social acceptance. Activity items, such as participating in a sport or helping a charity, had low alpha coefficients and so were removed. The remaining items are all largely subjective (e.g. I feel terribly alone and isolated; I have felt unsafe to walk in the neighbourhood; I have felt accepted by neighbours).

Conclusion As the authors indicate, it is not known whether the whole construct can be represented by these dimensions and, as the results of the concept mapping in the present study suggest, these dimensions are confined to individual and subjective, almost exclusively participatory or emotional, aspects of inclusion. It may therefore be very suitable for use in a limited range of settings, but may be less comprehensive and useful in large-scale studies requiring sophisticated analysis of multiple indicators.

Perceived community cohesion (Lev-Wiesel 2003)194

A number of researchers who represent the subjective approach to cohesion have proposed theoretical definitions and, despite the use of some different elements, share a common core meaning. Gross and Martin195 suggest that ‘perceived cohesion’ is the ‘sticking togetherness’ of the group, which later Hetcher196 and Bhattacharyya197 refer to as ‘solidarity’ (a deeply shared identity and shared values). Bollen and Hoyle198 define perceived cohesion as an individual’s sense of belonging to a particular group and his/her feelings of morale associated with membership of the group. A sense of belonging comprises cognitive and affective elements. Others replace the term ‘perceived cohesion’ by the term ‘psychological sense of community’ (Unger and Wandersman,199 Gruber and Shelton,200 Buckner201). McMillan and Chavis43 propose four elements of a psychological sense of community: membership (a feeling of belonging); influence (the power of member and community to affect each other); sharing values (fulfilment of an individual’s values by a community); and a shared emotional connection (based on a common history). Findings are reported for a sample of 215 residents (77 men and 138 women) of three different communities according to their social–economic structure: kibbutz (n = 39), village (n = 76) and urban centre (n = 100).

The questionnaire consisted of six parts (67 items) according to the elements constituting ‘perceived community cohesion’. A sense of belonging was addressed by 19 items (e.g. community members can count on me, I feel a part of the community). Social ties measures of the quality and quantity of social interaction were addressed by five items (e.g. I would like to have more friends in my community, I would like to meet community members more often). Perceived social support to measure individual belief in potential social support had 13 items (e.g. I feel responsible for my neighbours, I share my feelings and thoughts with my neighbours). Solidarity measures the feeling of togetherness and had 11 items (e.g. a member in trouble should turn to help to the leaders in our community, I contribute some of my time to help achieve community goals). Nine items related to rootedness measured a sense of place (e.g. I can’t see myself living in another place, I would like my children to live here when they grow up). Finally, 10 items measured feelings of alienation in a given community (e.g. the relationships among members of this community get worse, you cannot trust anyone in this community). Participants were asked to complete four-point scales to describe their own feelings, with response options ranging from ‘total agreement’ (4) to ‘total disagreement’ (1). Reliability was 0.87 (Cronbach’s alpha). To examine the extent to which the element measures (sense of belonging, social ties, social support, rootedness, solidarity and alienation) serve as potential indicators of ‘perceived community cohesion’ elements, a confirmatory factor analysis was performed. The minimal loading factor that had been set by the authors was 0.40; therefore, 15 items were dropped out of the questionnaire. The final questionnaire consists of 16 items that examined a sense of belonging, seven items that examined alienation, nine items that examined solidarity, seven items that examined social support, eight items that examined rootedness, and five items that examined social ties.

Conclusion The weakness of this research seems to derive from the fact that the elements were highly intercorrelated. This may be an indication that in reality they cannot be differentiated entirely and so it is not certain exactly what is being measured and no comparative measures were used.

De Jong Gierveld and van Tilburg (2006)202

Loneliness is an indicator of social well-being and pertains to the feeling of missing an intimate relationship (emotional loneliness) or missing a wider social network (social loneliness). The 11-item De Jong Gierveld Loneliness Scale has proved to be a valid and reliable measurement instrument for overall, emotional and social loneliness, although its length has sometimes rendered it difficult to use in large surveys. In this study, the authors empirically tested a shortened version of the scale on data from two surveys (N = 9448). Confirmatory factor analyses confirmed the specification of two latent factors. Congruent validity and the relationship with determinants (partner status, health) proved to be optimal. The six-item De Jong Gierveld Loneliness Scale is a reliable and valid measurement instrument, overall, for emotional and social loneliness, and is suitable for large surveys. The alpha coefficients for the six-item loneliness scale varied between 0.70 and 0.76 for the total adult population, indicating a quite reliable scale. As expected, the reliability coefficients for the three-item emotional loneliness scale were lower, varying between 0.67 and 0.74.

It is unknown whether the functioning of the shortened six-item loneliness scale, without the other five items, differs from the functioning of the six-item scale when it forms part of the 11-item loneliness scale. This study has also shown that the six-item loneliness scale and the three-item emotional and social subscales of loneliness are good measuring instruments for the broad age range of adults (18–99 years), as well as for the three age subgroups investigated: those aged < 45 years, those aged 45–64 years and those aged ≥ 65 years.

Conclusion Some of the scales have moderate alphas, and the measure does not appear to have been widely used. It is doubtful whether or not it can form a constituent part of a social inclusion measure, as it reflects the absence of relationship and wider networks. It is not clear whether or not the obverse of the loneliness score actually constitutes an inclusion index. Also, the items seem to be mood related and the authors do not seem to have explored the relationship with depressed mood, which might explain some of the variance of the measure. These objections seem sufficiently substantial not to warrant examination of the other major US loneliness measure (Russell DW, Peplau LA, Cutrona CE. The revised UCLA Loneliness Scale: concurrent and discriminant validity evidence. J Pers Soc Psychol 1980;39:472–80).

The participation scale (Van Brakel et al. 2006)68

The participation scale is a measure of participation, developed in Nepal, India and Brazil, using standard methods. The instrument was based on the participation domains of the International Classification of Functioning, Disability and Health (ICF), and assesses client-perceived participation. The participation domains of the ICF are learning and applying knowledge; general tasks and demands; communication; mobility; self-care; domestic life; interpersonal interactions and relationships; major life areas and community; and social and civic life.

Respondents rated their participation in comparison with a ‘peer’, defined as ‘someone similar to the respondent in all respects except for the disease or disability’. An 18-item instrument was developed in seven languages. Cronbach’s alpha coefficient was 0.92, intra-tester stability 0.83 and inter-tester reliability 0.80. Discrimination between controls and clients was good at a participation score threshold of 12. Responsiveness after a ‘life change’ was according to expectation. Each centre was to aim to enrol 90 subjects: 30 were reinterviewed to test inter-interviewer reliability and another 30 to test interviewer stability over a period of 1 month. The remaining 30 were interviewed by the expert and also had a participation scale interview. Fifteen of these were to be reassessed after 9–12 months to evaluate the dynamicity of the scale. Of these, 10 were to be subjects expected to experience a major life change in this period (e.g. clients who were to receive major rehabilitation assistance) and five were to be control subjects. In addition, each centre interviewed 10 control subjects without leprosy, disability or other significant health condition.

As no other validated participation assessment tool existed, the results were validated against the opinion of an expert – someone considered able to assess the severity of participation restrictions based on an interview. The experts rated the severity of participation restrictions on a 1–5 scale (1 = none, 5 = complete restriction).

Lelieveldt (2004)203

The data on the relationship between social capital and neighborhood-orientated forms of participation are derived from two surveys that were conducted in the spring of 2002 and 2003, in three OBAZ [Onze Buurt aan Zet (Our Neighbourhood’s Turn)] neighbourhoods (De Laares, Velve-Lindenhof and Bothoven) that were offered 3.5M guilders to improve the neighbourhoods, and three other neighbourhoods (Pathmos, Twekkelerveld and Deppenbroek) that were not. In 2002, 246 community respondents were drawn at random, which led to 144 completed interviews (response rate 58.5%), whereas in 2003, 336 addresses were drawn, which resulted in 163 completed interviews (response rate 48.5%). The empirical analysis was based on the combined total of 307 respondents. A neighbourliness scale was constructed on the basis of six items (including borrowed tools, talked about personal problems, helped each other, and quarrelled) and had a Cronbach’s alpha coefficient of 0.79. In the survey, attitudinal social capital was measured using a battery of seven items with statements about respondents’ own attitude towards the neighbourhood as well their perceptions of the attitudes of fellow residents. Neighbourhood residents were asked how often they kept an eye on children playing outside, kept an eye on neighbours’ homes, complained to people displaying annoying behaviour or put up a note with such complaints, removed dirt from the streets and, finally, how often they maintained public gardens or their own street-side garden. The six items yield a scale with a minimum of 0 (never) and a maximum of 3 (often) (α = 0.64). A second indicator of participation is based on an inventory of six categories of problems that respondents may have experienced and a simple ‘yes or no’ follow-up question that asked whether or not the respondent has tried to do something about the problem. The broad formulation of the follow-up question – ‘doing something in response to a problem’ – captures all kinds of participation, from directly approaching the person who caused a problem to more indirect forms such as contacting the police or civil servants. The third form of participation consists of voting at the latest municipal elections and has been included as a proxy for all kinds of institutionalised forms of political participation.

The structural dimension of social capital as measured by neighbourliness has the biggest and most consistent impact on participation (although the way these are measured suggests that they are probably confounded), followed by sense of duty and finally by trust, which turns out to be non-significant for informal governance and negatively related to the conversion of problems into action.

Conclusion The internal consistency of some scales is poor, and the results may have been achieved because different measures may have been confounded. The study presents some interesting ideas about participation activity, but uses only voting for civic engagement. Probably too limited to be useful.

The Impact on Participation and Autonomy Questionnaire (Sibley et al. 2006)204

As the title of this measure implies, it is concerned with the assessment of the impact of disability, illness and handicap on autonomy and participation. It is doubtful, therefore, how relevant the measure would be in a general population context. The authors, in Nottingham and Southampton, aimed to evaluate the validity and reliability of an English version of the Impact on Participation and Autonomy Questionnaire (IPA). The original Dutch IPA has been shown to load on to five factors. Subjects were 213 people with multiple sclerosis, rheumatoid arthritis, or spinal cord injury and general practice attendees, stratified by level of disability (median age 54 years: 42% male, 58% female). Self- and interviewer-administered outcome measures. Other measures: SF-36, London Handicap Scale, three domains of the Functional Limitations Profile (FLP) (household management, social integration, emotion).

Respondents are asked to ‘rate their chances of …’, for instance ‘getting around in my house when I want’; ‘visiting relatives and friends when I want’; ‘going on the sort of trips and holidays I want to’; ‘getting washed and dressed the way I wish’ as very good, good, fair, poor and very poor.

The scale has good psychometric properties. Confirmatory factor analysis confirmed the construct validity of the IPA (Normal Fit Index > 0.98, Comparative Fit Index > 0.99), indicating a good fit to the model. Convergent and discriminant validity were confirmed by the predicted associations, or lack of, with the exception of a poor association between the ‘social life/relationships’ IPA subscale and FLP–Emotion. Internal reliability of the IPA was confirmed (Cronbach’s α > 0.8; item-total correlations for all subscales > 0.5). Test–retest reliability was confirmed for all items (weighted kappas > 0.6) and subscales (intraclass correlation coefficients > 0.90). The authors suggest that further research is required to examine the responsiveness of the IPA to change over time, its clinical utility and suitability for use with people from ethnic minorities and with older people.

Conclusion The five factors are (a) autonomy indoors; (b) family role; (c) autonomy outdoors; (d) social life and relationships; and (e) work and education. However, the questions are much more focused on the former than the latter (15 items in ‘a’ and ‘b’): money and leisure only one each; relationships (7); work and education (6). Cronbach’s alpha coefficients are highest for indoor and outdoor autonomy. This suggests that this measure is more useful where the focus on handicaps and limitations to autonomy and participation – but, even so, participation is not actually assessed in much detail.

Mokken scales

Mokken scales are similar to Guttman scales but they are probabilistic, whereas Guttman scales are deterministic. That is, in Mokken scales a respondent answering an item positively will have a significantly greater probability than null to answer a less difficult item in a positive way as well, whereas in perfect Guttman scales answering an item positively means that the respondent will answer all less difficult items positively also.

All items in a Mokken scale have different difficulties, as reflected in different proportions of positive responses. The graphic representation (called a trace line) of the probability of a positive response to an item should increase monotonically as the latent trait increases along the x-axis (and where the y-axis, of course, is the probability). Double monotony must not exist (that is, trace lines of items in a scale should not intersect). Also, trace lines must be steep enough to produce only a limited number of Guttman errors (exceptions to the rule that a positive answer to an item implies a positive answer to all easier items). Loevinger’s H measures the conformity of a set of items to Mokken’s criteria and validates their use together as a scale of a unidimensional latent variable.

Loevinger’s H is based on the ratio of observed Guttman errors to total errors expected under the null assumption that items are totally unrelated. Let E = the probability of a Guttman error and let Eo equal the same probability under the null model of totally unrelated items. H = 1 – E/E_o, as discussed below.

Let item j be easier than item i, which in formulaic expression means that P(X_j = 1) > P(X_i = 1) – the probability that j is 1 is greater than the probability that i is 1. Then H_ij = l – E/E₀, where E = P(X_i = 1,X_j = 0) and E₀ = P(X_i = 1)  × P(X_j = 0) for a random subject. When there are no Guttman errors, H_ij = 1. When the response is random (the null model), H_ij = 0. (Of course, when computing these values one must recode where necessary so that the 1s and 0s have a consistent meaning across items.)

See also Streiner and Norman (2008: 323–4).

Correlations

Spearman’s rho is a form of rank order calculation. It is calculated with the same formula as for Pearson’s r correlation, but using rank rather than interval data. As scales are rank-order data, rho is often used in place of r when correlation is called for, as in inter-item correlations, below. The median rho between all pairs of items in a scale is a classic measure of reliability, in the sense of internal consistency. Rho values > 0.60 are considered to be necessary for an adequate scale.

Interval and ordinal scales

When may ordinal scale data be used in regression and other interval techniques? Technically, never.

As an independent Methodologists use a rule-of-thumb that there must be a certain minimum number of classes in the ordinal independent [Achen (1991) argues for at least five; Berry (1993: 47) states five or fewer is ‘clearly inappropriate’; others have insisted on seven or more]. Use of seven-point scales or higher would seem best, but it must be noted that use of five-point Likert scales with interval procedures is extremely common in the literature.

As a dependent One method is to test to see if there are significant differences in the regression equation when computed separately for each value class of the ordinal dependent. If the independents seem to operate equally across each of the ordinal levels of the dependent then use of an ordinal dependent is considered acceptable.

Can Likert scales be considered interval?

Likert scales (e.g. strongly agree, agree, etc.) are very commonly used with interval procedures, provided the scale item has at least five and preferably seven categories. Most researchers would not use a three-point Likert scale with a technique requiring interval data. The fewer the number of points, the more likely the departure from the assumption of normal distribution, required for many tests. Here is a typical footnote inserted in research using interval techniques with Likert scales: ‘In regard to our use of (insert name of procedure), which assumes interval data, with ordinal Likert scale items, in a review of the literature on this topic, Jaccard and Wan (1996: 4) summarize “for many statistical tests, rather severe departures (from intervalness) do not seem to affect Type I and Type II errors dramatically” ’.

Likert scales are ordinal, but their use in statistical procedures assuming interval-level data is commonplace for the reason given above. Note, though, that under certain circumstances, Likert and other rank data can be interval. This would happen, for instance, in a survey of children’s allowances if all children in the sample got allowances of $5, $10 or $15 exactly, and these were measured as ‘low’, ‘medium’ and ‘high’. That is, intervalness is an attribute of the data, not of the labels. In most cases, of course, Likert and rank variables are ordinal but the extent to which they approach intervalness depends on the correspondence of the ordinal labels to the empirical data.

According to Streiner and Norman (2008), this debate ‘shows no sign of resolution’ (p. 53) and ‘it appears that under most circumstances … one can analyse data from rating scales as if they were interval without introducing severe bias’ (p. 53).

Measuring change

Websites

http://faculty.chass.ncsu.edu/garson/PA765/standard.htm

http://faculty.chass.ncsu.edu/garson/PA765/datalevl.htm

Background to the study

The relevant literature has been summarised in the Phase I report (Huxley et al. 2006) and will not be repeated here.

The Phase I project was in four parts.

1. An update of the social exclusion and mental health review undertaken at London School of Economics (LSE), and a parallel review of social inclusion and its measurement.

2. A review of existing measures, whether derived from the formal reviews above, web searches or personal contacts.

3. Nine concept mapping exercises, conducted with differently composed groups, including mental health service users, professionals, and members of the general population.

4. Review of the findings and this report by experts. One expert group consisted of the authors of the report, and the other of the National Institute of Mental Health England (NIMHE) social inclusion research and evidence coalition.

Prior to conducting our Phase I project we consulted with the National Co-ordinating Centre for Research Methodology (NCCRM) to ascertain whether any new measure should be applicable only to people with mental health problems or for a broader constituency. We were told that it should be usable in both contexts and so our project worked on the basis that a general measure, applicable in mental health service contexts (also interpreted widely) would be the most desirable end product. We are assuming that the Phase II call for proposals retains this objective, as it refers to the purpose of the measure being for ‘routine outcome measurement’ as well as ‘mental health research’. We believe that to limit such a measure to mental health research only will contribute to the isolation of such research from that conducted in other clinical and non-clinical contexts, prevent comparative research between different social and clinical groups of respondents, and constitute less value for money than a more general measure of wider applicability. It is also the case that the extent or degree of inclusion or exclusion of any particular group in society cannot be estimated without reference to the same data in the general population. Limiting the development of an index of exclusion to people in receipt of secondary care psychiatric services only would be to commit a similar error to the development of a psychiatric nosology based on secondary care cases only (Goldberg and Huxley 1990) and could result in a partial, incomplete or even mistaken picture.

Phase I findings

• Social inclusion is widely agreed to be:

  1. – relative to a given society (place and time)

  2. – multidimensional (whether those dimensions are conceived in terms of rights or key activities)

  3. – dynamic (because inclusion is a process rather than a state)

  4. – and multilayered (in the sense that its causes operate at individual, familial, communal, societal and even global levels).

• Any measure of inclusion needs to be able to accommodate the above characteristics.

• We identified two broad approaches to the measurement of social inclusion: (1) social indicators of inclusion and (2) perceived inclusion measures. The proposed measure should encompass both aspects.

• The reasons for wanting to use an index of inclusion are variable, and include policy, theory, practice, evaluation, research and ‘real-world’ uses. It may be that one index that fits for all of these purposes cannot be created, and it may be that the indicator approach is better for policy purposes and the perceived inclusion approach for clinical and ‘real-world’ purposes. It may be that for other purposes some combination of both approaches might be valuable and acceptable, but this has not been tested to date.

• We found that there were several existing social indicator measurement exercises that might be used to produce a social indicator based measure of social inclusion. We cite the main candidates for this in the main Phase I report (Huxley et al. 2006).

• Another option (parallel to the suggestion of Levitas et al. regarding an exclusion index) is that there should be a social inclusion module incorporated into a UK longitudinal household survey. The disadvantage of this is that household surveys may miss the most excluded groups. The development of a new measure would permit this option to be explored subsequently.

• A further option is that an index might be garnered from existing UK social surveys, and Levitas et al. identified the most promising ones for young people, older people and adults of working age. One disadvantage of this approach is that the wording of the questions is predetermined and may not be the most fit for purpose or acceptable to respondents.

• A disadvantage of all the indicator approaches is that it is often unclear why certain indicators have been chosen above others, by whom they were chosen, and that the form and content of the questions used may not be acceptable to the respondents.

• On the basis of the evidence we have reviewed we suggest that any proposed social inclusion index should focus on the availability of opportunity to access material and other resources, and the subjective view of this availability. It should address the extent to which the person participates in these life activities, and also the person’s subjective perception of the value or benefit of these activities for themselves. It should also assess the degree to which the person wishes to have more or less or the same level or type of participation in each life domain, otherwise it is not possible to encompass the feelings of those who are satisfied with what might be a low level (personally and normatively) of activity through choice. While everyone in society has the right to participate in a way that facilitates social inclusion, individuals have different needs for access and participation; therefore an index of social inclusion needs to capture individual perception of inclusion rather than simply imposing standards from outside.

• There is no single, tested and robust measure of social inclusion of this sort that is an accepted standard measure. Measures of the component parts of a possible measure, such as access to material goods, etc., and social and community participation, are available, but usually do not have adequate psychometrics, with the exception of the Van Brakel measure, and there are some reservations about its use. The instrument developed by Wilson, currently being applied in North Adelaide, might prove to contain some useful scales, but the results have not been reported yet. Wilson has agreed (personal communication, 23rd January 2007) to the sharing of information and possible research collaboration).

• We are therefore left with the need to create a new index, as suggested in our draft Phase II proposal submitted with our Phase I proposal. We think that it is worthwhile attempting a combined approach, rather than leaving social indicator research to one group of investigators and perceived inclusion research to another.

Sample sizes

This project involves the creation of a new instrument and so we have no data upon which to base power calculations. However, we have based sample sizes on our previous experience of measure development, and what we know is achievable in the time available. Where possible we have based sample sizes on power calculations from previous instruments of known relevance to social inclusion, such as some of our quality-of-life scales, and categorical variables such as the rates of unemployment.

Component one: selection of questions and wording

The NIMHE social inclusion coalition, including service users and carers, will meet to consider questions in each of the domains identified in the Phase I project (see Table 1). Candidate questions and wording will be reviewed and consensus reached on a draft version; previous work by members of the NIMHE social inclusion coalition (Bates, Davis, Secker and Huxley) has produced a number of different options for questions. Part of this process will involve recontacting some of the members of our original concept mapping groups and also those organisations responsible for the development of social indicator measures from existing survey data (identified in the Phase 1 report) to obtain their opinions and feedback, for example Office of National Statistics (ONS).

Without prejudging the actual questions to be used (as indicated earlier, there are several different, albeit incomplete options available), Bates has provided some pertinent examples, which fit into this style of enquiry (see below).

• Are you working?

• Is work easily/readily available?

• How many hours a week do you work?

• How do you feel about the availability of work? (delighted–terrible)

• How do you feel about the work you do? (delighted–terrible)

• Would you like to work more or fewer hours than you do, or change your job?

• Do you think that you are generally safe where you live?

• Do you walk around safely in the day?

• Do you walk around safely at night?

• How do you feel about your neighbourhood safety? (delighted–terrible)

• Would you like the area to remain the same or be more safe, or would you like to move away?

• Are you making use of any type of educational opportunity (classes, etc.)?

• Are educational opportunities readily available to you?

• How often are you making use of them?

• How do you feel about the opportunities available? (delighted–terrible)

• How do you feel about your educational activities? (delighted–terrible)

• Would you like to take up more educational opportunities, remain the same, or do less?

It is perhaps worth emphasising at this point that the delighted–terrible scales for the assessment of subjective items continue to be widely used both in the UK and the USA in community mental health studies (e.g. most recently in Nelson et al. 2007). This enables data to be accumulated on the same measuring scale and domain items and results from different services and research projects to be compared by location and over time.

Component two: pre-testing (larger version)

The draft instrument will be tested in a convenience sample of 30 people aged between 18 and 65 years, including those with and without mental health problems, and balanced for gender and suitably profiled for age. People with mental health problems will be identified in association with our partners (NIMHE social inclusion coalition members). All participants will be provided with information about the study at the time of request to participate, and it will be explained that participation is voluntary, will not affect the provision of any services that they may be using, is confidential, and that ethical approval and research and development approval has been sought from the appropriate authorities. The purpose of the pre-testing is to clarify ambiguities in question wording (using cognitive appraisal techniques to assess question wording, together with further questions concerning aspects of face validity, content validity and respondent burden), to confirm the appropriateness of response scales, to determine acceptability and to estimate completion time. We successfully undertook a similar exercise in the development of the Quality of Life in Later Life (QuiLL) assessment. Participants will be interviewed in their own homes, unless they prefer to be interviewed elsewhere. Appropriate modifications will be made to the questionnaire, on the basis of this exercise, to produce a preliminary field test version.

Component three: field testing (longer research version)

This version of the instrument will be tested in a sample of 250 people, including mentally healthy people in the general population, and people with CMDs. This sample will be drawn from the general population.

The preliminary field-testing is to determine item response rates and acceptability in the group as a whole, and in different mental health status groups. Accordingly, a further sample of 50 MHSUs will be assessed using this version of the instrument (sourced from either SLAM or the NIMHE social inclusion coalition members, or Hafal in Wales). We will examine the data for systematic differences in the response to the questionnaire between groups.

Component four: item selection and data reduction

On the basis of component three data we will select items for the final version of the questionnaire. Standard psychometric techniques for item analysis and selection criteria will be applied to preliminary field test data as part of the instrument testing and refinement phase, including:

1. evaluation of discriminatory power and frequency of endorsement of items; items endorsed by more than 90% and fewer than 10% would be excluded but will need to be carefully applied in the different population subgroups; for example, if it is above 10% in the general population, but below 10% in the mental health sample we would not necessarily discard the item as it may be important

2. determination of item overlap and/or redundancy, examination of the intercorrelation between items

3. assessment of homogeneity of items

4. item convergent and discriminant validity analyses

5. evaluation of the reliability and validity of individual items.

Further data reduction will be undertaken using exploratory factor analysis and item response methodology. See the analysis section for the justification for the use of item response theory for internal scaling and Appendix A for more technical details.

Component five: final field testing and psychometric testing of the final instrument (short operational version)

Final field test data will be analysed cross-sectionally and longitudinally, as part of an evaluation of the psychometric properties of the questionnaire, including its reliability, validity and responsiveness. The final version of the questionnaire will be evaluated according to the gold standard criteria for reviewing instruments used by the Medical Outcomes Trust (Lohr et al. 1996). At this stage the questionnaire will be assessed in respect of: conceptual and measurement model, reliability, validity, responsiveness, interpretability and burden. The analysis of reliability, responsiveness and validity is described in detail in the analysis section. In order to test the generalisability of the index and ability to capture the extent of inclusion in excluded groups we propose to assess a group of younger people with arthritis, through a project already under way at LSE (this will form part of the LSE contribution to Phase II).

Component six: beta-testing

A training package will be developed for (and with) mental health service practitioners, to inform understanding of how to conduct standardised research interviews, how to use the instrument, how to analyse, interpret and present results, and how the results might be used to inform practice. Practitioners involved in the NIMHE social inclusion programme (social workers, nurses, psychiatrists, psychologists and others working in mental health teams) will be offered this training and asked to use the social inclusion index routinely in their day-to-day practice, collecting data on new referrals to each service providing some intensive form of social care. Data entry and analysis services will be offered to participating agencies by the Centre for Social Carework Research at Swansea.

Alternate forms – different cultural, ethnic and language adaptations (with the exclusion of a Welsh version produced by University of Wales, Swansea, at no cost to the study) – will not be tested at this stage, as it is a costly process, which is beyond the scope of this study, given the available resources; further adaptation will be a necessary future development of the work (but see below for the developing international collaborations). Similarly, a self-response version of the long and short versions would require resources beyond the present study, and alternative sources of funding are being investigated by the applicants.

Component two

Pre-testing data, including questions about respondent burden will be analysed using descriptive statistics.

Component three

As we are interested in both the theoretical solution using all the instruments applied in the survey we will use factor analysis at this stage. The substudy analysis is intended to ascertain whether there are different constructs – such as participation, social inclusion and social capital – or whether they are all based on a single underlying factor.

In addition, between-group differences in objective and subjective indicators of social inclusion (e.g. mental health status – problem/no problem; age; gender, etc.) will be examined using chi-squared statistics for categorical variables and independent samples t-tests or analysis of variance (ANOVA) statistics for continuous variables. The association between objective and subjective indicators of social inclusion will also be assessed, using correlation coefficients, chi-squared statistics and independent samples t-tests or ANOVA statistics. Non-parametric equivalent tests (Mann–Whitney, Kruskal–Wallis, Spearman correlation) will be applied if the distribution of samples is skewed or if sample sizes are inadequate for parametric tests. Skewness of data will be assessed using the following formula: skewness/standard error of skewness = ± 1.96.

Component four

As in the development of the QuiLL (Evans et al. 2005), the first stage of data reduction will involve a psychometric evaluation of the properties of the instrument. This process will establish the reliability of individual items, item overlap, redundancy, frequency of use and missing items. The internal consistency of the resulting scales will be examined using Cronbach’s alpha coefficient to identify scale items that can be removed without detracting from the psychometric properties of that scale.

Factor analysis will then be undertaken in order to reduce the number of questions by eliminating those that fail to load on the key factors. Suitability of the data for factor analysis will be assessed by Bartlett’s test of sphericity (which should be significant, p < 0.05) and the Kaiser–Meyer–Olkin measure of sampling adequacy (which should be a minimum of 0.6). The decision on the number of factors to retain will be based on Horn’s parallel analysis, using Monte Carlo Principal Component Analysis.

As traditional psychometric analysis such as factor analysis assumes an interval level of measurement,131 this might not be the most appropriate method for item reduction and scaling. Item response theory models, which have been developed for variables measured at a nominal or ordinal level, might offer a more sophisticated and comprehensive approach to detecting scales within the social inclusion index. Since the index is likely to include variables measured at nominal, ordinal and interval levels, we will use both methods of analysis, details of which are given below. This approach is justified because the categorical measures can be entered as 0,1 into the factor analysis, and the continuous variables such as the subjective rating scales can be dichotomised for use in item response testing.

Further technical details about item response theory are given in Appendix A.

Component five

Information and instructions

Potential uses

Figure 7 is an example of changes taking place in subjective ratings for one individual who was moved when his hostel was closed, and whose situation deteriorated when in a new group home. Based on this deterioration in his scores, he was placed with a foster family and his scores all dramatically improved, with the exception of his ratings for his own family, which remained constantly poor. No intervention in relation to his own family was taking place.

FIGURE 7.

Routine outcome assessment in life domains.

One-off assessments

As part of a clinical assessment package, the SCOPE could be used to assess the individual’s circumstances in respect of inclusion and participation, and where if at all they need some form of assistance to achieved desired changes.

Visual feedback

Graphics of the sort in the figure above can be produced for individuals. This can be done very simply using an Excel spreadsheet. Enter scores in two rows above each other, for Time 1 and Time 2 responses on the delighted–terrible scale (1–7) by domain, and then highlight the scores in both rows and press the ‘Chart Wizard’ button and select the presentation style that you want.

Comparison with other groups and local population

In many instances the national surveys can be broken down to locality level, and so it becomes possible to compare the inclusion levels of service user samples to the local population figure, for example from the Census data.

Here are some SCOPE comparisons between a deprived community in North West England with national data from the NPMS. Satisfaction with leisure activity in the area sample is significantly lower than the national average.

Local area sample (national average)

• 80/100 feel leisure opportunities are restricted (64%)

• 83/100 want a more active social life (62%)

• 47/100 want to participate more fully in family activities (28%)

• 56/100 not a member of community groups (47%)

• Satisfaction with leisure activity 3.7 (4.3) (p < 0.001).

Here are some further comparisons between a service user sample and the comparable figures in the local area.

Service user sample (local population averages)

• 5% employed compared with 61%

• 24-hour working week compared with 38 hours

• 53% seen friend last week compared with 80%

• 57% have a close friend compared with 95%

• 16% contact + relatives less than monthly 30.%

As some of these items will vary by gender and age, make sure that, where available you disaggregate the data to permit age group and gender comparisons.

This shows the low employment levels in service user populations compared with the local level, and, for those in work, fewer hours worked and at a lower rate of pay (not shown).

The following histogram (Figure 8) shows the differences between the SCOPE scales for healthy people with no illness (general population), people with severe mental health problems (SMI) and people with CMD (anxiety and depression).

FIGURE 8.

Subjective scales by mental health group.

Social and Community Opportunities Profile and related publications (in the public domain or available from the authors)

Evans S. Social Inclusion and participation: a reanalysis for common mental disorder and mentally healthy groups using IOP survey data. Final report to Social Exclusion Unit. London: Institute of Psychiatry (Social Work and Social Care Section, Health Services Research Department); 2004.

Evans S, Huxley P, Webber M, Davidson B, King S. Social inclusion and mental health: policy, practice and evidence from the UK. In Proceedings of The Mental Health Services Conference 2005. Sydney: The Mental Health Services Conference Inc; 2006.

Evans S, Webber M, Huxley P. Social Inclusion – the evidence. In Proceedings of The Mental Health Services Conference 2005. Sydney: The Mental Health Services Conference Inc; 2006.

Huxley P, Evans S. Social inclusion and mental health: development of measures for use in research and operational practice. The Baptist University, Hong Kong, 11 December, 2006.

Huxley P, Evans S. Social inclusion: a concept mapping exercise. The Mental Health Services Conference (TheMHS), Melbourne, 4 August 2007.

Huxley P, Evans S. Social inclusion and mental health: experiences from the UK. School of Sociology & Social Work, University of Tasmania, August 2007.

Huxley P, King S, Davidson B, Evans S. Promoting social inclusion through mental health service provision. In Proceedings of The Mental Health Services Conference 2005. Sydney: The Mental Health Services Conference Inc; 2006.

Huxley P, Webber M, Evans S. Measuring social inclusion. In Proceedings of The Mental Health Services Conference 2005. 2006.

Huxley P, Evans S, Munroe M, Philpin S. SCOPE For social inclusion. A Life in the Day 2008;12:33–5.

Plenary Address: The Promotion of mental health through social inclusion. TheMHS Conference, Sydney, 22 August 2002.

Plenary Address: Looking toward excellence: the measurement of social inclusion in mental health services. The Mental Health Services Conference (TheMHS), Melbourne, Australia, August 2007.

Frequently asked questions

Are translations available for minority groups?

• Welsh translations are available. No other translations have been undertaken. We have been working with an Islamic community support service to develop a version for use in Islamic communities in the UK. At present this version is, on the advice of the service managers, in English, and has one additional section on beliefs and their importance to the individual.

How suitable is this instrument for persons with SMI?

• It has been designed and used in a mental health context and is suitable for use with people who have severe illness. It should not be used when the person is in an acute phase of illness with severe symptoms of psychosis or clinical depression.

• No adverse events or reactions have been documented during the course of using it with clients of community mental health teams, or independent sector mental health services.

Are the questions understandable and do they make sense to homeless persons?

• In most instances homeless people are in some form of shelter. The questions are understandable and can be used in this context. People living on the street could answer the accommodation questions.

Are the questions understandable and do they make sense to those with severe cognitive impairments?

• No, we have not tested the SCOPE on people with severe cognitive impairment and would not expect them to understand many of the questions, or to be able to recall the answers if they did understand the question.

• From our previous experience in assessing the QoL of people with mild to moderate dementia, (using some of the same SCOPE questions) we have found that they are able to understand and express an opinion. We wrote a paper entitled, ‘Just because I don’t know where I am doesn’t mean I don’t know what I like’.

Are the questions understandable and do they make sense to patients recently discharged from psychiatric hospitals?

• The original QoL work, which used the same life domain structure and the same SWB scale, was devised for people who were long-stay hospital patients, most of whom returned to the community. Many of the study samples used to develop SCOPE were people discharged from hospital to the care of community teams, so we would be confident that this group would have no problem with the SCOPE.

Can it be used in community surveys?

• The short version has only been tested to date in surveys of students and people with mental health problems. Please contact us if you are planning a community survey.

• The original long research version could be used should you want to compare your community sample results with ours, and with the normative data from the original surveys.

How do different versions differ?

• The Islamic version has a section on belief, which follows the same structure as the other life domains.

Are you able to offer help in analysing my data?

• If you supply us with the details we need we can produce you results in a format to suit you. We would also calculate how much of the change observed in your sample, can be said to be reliable change using appropriate estimates. We have a brief Powerpoint presentation available for those wishing to calculate reliable change (in group data) for themselves.

• We would not charge for this service but would ask to keep an anonymised set of your data to add to our SCOPE databank.

Where can we get hold of you and SCOPE?

• Sally Madge

• Peter Huxley

• Sherrill Evans

• Karen Evans

Contact details:

• Postal address: Centre for Social Work and Social Care Research, College of Human and Health Sciences, Swansea University, Singleton Park, Swansea SA2 8PP, UK.

Scoring and analysis

Although there are several ways that the SCOPE can be broken down into scales, we recommend that the following scales have the best psychometric properties.

• overall inclusion – this is a single figure on the delighted–terrible scale of 1–7

• Perceived Opps scale – sum of items #

• SatOpps scale – sum of items #

• participation scale – 013 items

• all of the objective items are single figures and should not be added into a scale, but treated independently – 014 items

• to measure change over time the scales should be compared using paired t-tests, and the O-variable and P-variable by chi-squared analysis.

Appendix: examples of information sheets and consent forms for research purposes