Outcomes of social care for adults: developing a preference weighted measure

Article history

The research reported in this issue of the journal was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA programme in April 2007 under the newly established NIHR Methodology Panel. The HTA programme project number is 06/96/01. The contractual start date was in August 2008. The draft report began editorial review in March 2011 and was accepted for publication in August 2011. The commissioning brief was devised by the NCCRM who specified the research question and study design.The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

none

Copyright statement

© Queen’s Printer and Controller of HMSO 2012. This work was produced by Netten et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This journal is a member of and subscribes to the principles of the Committee on Publication Ethics (COPE) (http://www.publicationethics.org/). This journal may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NETSCC, Health Technology Assessment, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2012 Queen’s Printer and Controller of HMSO

Objective

The aim of this study was to develop a measure of social care outcome, an equivalent to the quality-adjusted life year (QALY) in health, which could be used in a range of circumstances.

Design

The project drew on previous and parallel work developing the Adult Social Care Outcome Toolkit and the national Adult Social Care Survey. We developed and tested an instrument designed to reflect service users’ social care-related quality of life (SCRQoL) and tested it with 30 service users from a variety of user groups and 300 older home care service users. In parallel, we explored discrete choice experiment (DCE) and best–worst scaling (BWS) approaches to preference elicitation with 300 members of the general population, and cognitively tested these with service users. We also cognitively tested a computer-aided time trade-off (TTO) exercise using SCRQoL attributes with members of the general population. In the second phase, using the finalised instruments, BWS interviews were conducted with 500 members of the general population, TTO interviews with a follow-up sample of 126 of these respondents, and BWS interviews with 458 people using equipment services.

Main outcome measures

The final measure had eight domains: personal cleanliness and comfort, accommodation cleanliness and comfort, food and drink, safety, social participation and involvement, occupation, control over daily life and dignity. In addition to measuring current SCRQoL, the instrument includes questions used to establish service users’ views of their ‘expected’ SCRQoL in the absence of services. The difference between a person’s current and ‘expected’ SCRQoL provides an indicator of service impact.

Results

There was good evidence for the validity of the descriptive system and the validity of the current, expected and SCRQoL gain scales. The DCE and BWS approaches yielded similar results and, once introductions made clear, were understood by service users. BWS was used for the main stages, as it had technical and cognitive advantages. The computer-aided approach to TTO worked well, and respondents found questions acceptable and understandable. There were no substantive differences in the preferences of service users and the general population. The key domain was control over daily life, with the lowest and highest levels strongly estimated in all models. After allowing for observable heterogeneity, service users’ preferences appeared to be more closely associated with their own SCRQoL than with those of the general population. The consistency of the results with the results of a previous study allowed the final model to be based on the preferences of 1000 members of the general population. A formula based on the relationship between TTO and BWS values was estimated for a social care QALY, with ‘0’ equivalent to ‘being dead’ and ‘1’ being the ‘ideal’ SCRQoL state. Members of the population experienced significantly higher SCRQoL than service users.

Conclusions

Although further work is needed, particularly to develop an equivalent measure for informal carers and to explore the links with health QALYs, the measure has considerable potential. A number of methodological advances were achieved, including the first application of TTO in a social care context and use of BWS to establish service user preferences.

Funding

The National Institute for Health Research Health Technology Assessment programme.

Background

There is increasing policy focus on outcomes in the field of health and social care in England. 1,2 This reflects a wider emphasis on the importance of reflecting outcomes of publicly funded services, which has been evident for some time, for example in reflecting productivity of public services in National Accounts. 3 In times of financial stringency it is particularly important to estimate the impact of different interventions accurately if we are to identify cost-effective alternatives and make the best use of limited resources. In the field of health, the quality-adjusted life year (QALY) is widely accepted as the measure of health outcome used to evaluate the cost-effectiveness of new health interventions. There is no such equivalent for social care, which has fundamentally different objectives to health care, making it difficult to assess accurately the impact of interventions that are closely aligned with, and can have direct impacts on, the demand for health care. The Wanless review4 used an early social care outcome measure to comment on future resource needs, but identified the need for much more development of outcome tools in this area. Ideally, commissioners and providers should have outcome information to draw on as a management tool for allocating resources to ensure good productivity and efficient use of those resources.

This report describes the results of a study designed to develop an equivalent measure to the health QALY, which reflects the particular characteristics of social care and can be used in a range of circumstances to reflect the impact and value of social care interventions. This first chapter describes the aims and objectives of the study, provides a broad overview of the design and describes the conceptual basis of the measure. In Chapter 2 we describe the basis for, and development of, the items that make up the instrument, and in Chapter 3 we outline how we tested the validity of the measure. Chapter 4 discusses how we tested a variety of approaches to preference elicitation and developed the methods used for the basis of the utility weights. The results of this work are reported in Chapter 5. The report ends with an overview of the final measure, discusses the potential uses and value of the measure, and suggests future work needed.

Aims and objectives

The aim of the study was to generate a preference-weighted measure of social care outcomes that could be used in a variety of circumstances, including social care evaluations, cost–utility analyses, weighted output of government-funded social care for National Accounts,3 and policy analyses.

The objectives were to:

• develop a measure of social care outcome that captures all relevant domains, has credibility in the social care community, has been cognitively tested and demonstrates good psychometric properties

• test a variety of approaches to establishing preferences for methodological soundness and practical application with the general population

• investigate the applicability of these methods to a service user population

• conduct a population preference study

• develop a set of weights for calculating social care quality-adjusted life years (SC-QALYs) and weighting measures of social care output

• explore how service users’ views differ from those of the general population, and establish alternative preference weights if appropriate

• identify the factors that affect preferences among the general population and service users.

Research design

The project comprised two phases. The first covered development and feasibility work to inform the design for a main phase in which preference weights for the final measure were derived from a study of both the general population and service users. The project undertook and drew on a number of different data collections, summarised in Table 1 and described below.

There were two strands to the development and feasibility work, conducted in parallel:

• the development and psychometric testing of the measure itself

• exploration of the feasibility of different approaches to establishing preferences.

The aim was to refine and develop the measure, while testing a number of elicitation approaches with the general population. The results of these exercises could then feed into testing the measure and piloting preference elicitation with service users. This allowed us to evaluate both the measure itself and the preference elicitation methods.

The development of the measure included conceptual development, a focused literature review, analysis of data sets that incorporated previous versions of the measure, cognitive testing of items with service users, and a survey of older people using home care services (see Table 1, and Chapters 2 and 3). The preference elicitation feasibility work involved testing alternative discrete choice approaches through a survey of the general population and cognitively testing these approaches with service users. In addition, we wanted to explore the feasibility of anchoring the measure to the state of ‘being dead’, permitting equivalent cost–utility analyses as those used in health care. To this end, we cognitively tested the use of time trade-off (TTO) techniques with respect to social care-related quality-of-life (SCRQoL) attributes with members of the general population (see Table 1 and Chapter 4).

We built on previous work in the field of outcome measurement in social care, and the project both contributed to, and drew on, other work that fed into the development of the Adult Social Care Outcome Toolkit (ASCOT) (www.pssru.ac.uk/ASCOT). The measure drew on earlier attempts to develop a measure of social care outcome for older people,5 which was then extended through a number of studies to be applicable to a wider group of social care service users. The extension to other groups was undertaken partly as a result of an interest in developing an approach to developing outcome-based quality weightings for publicly-funded outputs for the purposes of National Accounts,3 in which the Personal Social Services Research Unit (PSSRU) took the lead on developing an approach for Personal Social Services (PSS). 6 This work led to a Treasury-funded study led by the Office for National Statistics (ONS),7 as part of which the ASCOT was developed, drawing on a study of care homes,8 low-level services9 and a preference study. 10 The preference study drew on the development work undertaken as part of this study and provided a helpful test of the measure during its developmental stage (see Chapters 3 and 4). In addition, development work for the national Adult Social Care Survey (ASCS), which now includes the ASCOT measure, provided other opportunities for testing the measure (see Chapter 3). 11

The second phase drew on this iterative work and, for the final version of the instrument, involved a main preference study of the general population using the best–worst scaling (BWS) approach, and a follow-up TTO study with a subsample of this population. We also conducted a preference study of people who use equipment services, and a follow-up study with a subsample of these to test further the final version of the instrument (see Table 1 and Chapter 5).

An important element of the work was the involvement of local councils, and other authorities in the field were recruited as ‘critical friends’ to the project. In the early stages of the work, to assist in ensuring the acceptability of the approach being developed, they were invited to comment on our thinking with respect to the conceptual basis of the measure and were later kept informed on its practical application. 12 The councils also provided access to service users who were interested in participating in various stages of the study through the annual User Experience Surveys (UESs) that councils are required to conduct. 13 Service users were also consulted through the user and carer group at the PSSRU, London School of Economics (LSE).

Ethical permission was obtained from the Social Care Research Ethics Committee (SCREC) for all stages of the work that involved service users. The Association of Directors of Adult Social Services (ADASS) supported the study, and research governance approval was obtained from all participating local councils. For elements of the study that involved only members of the general population, ethical approval was obtained from the University of Kent ethics committee. All participants were given gift vouchers to thank them for their involvement.

Conceptual basis of the measure

Before we can develop a measure of outcome, we need to be clear about exactly what it is we are intending to measure. We have discussed the conceptual basis of outcome measurement in social care in more detail elsewhere. 14–16 Here, we summarise what we mean by social care, describe the implications of that for what we wanted to be able to reflect in our measure, and briefly consider the implications of the objective of a measure that can be seen as a SC-QALY equivalent.

Conclusion

The project was designed to establish a preference-weighted and ideally anchored generic measure of social care outcomes that could be used for a variety of purposes. The first task was to clarify exactly what it is we are attempting to measure. Drawing on the concepts identified above, in the next chapters we now turn to the development and validation of the instrument.

Introduction

As noted in Chapter 1, we did not start from scratch in developing the measure: we were building on previous work and indicators that had been developed specifically to measure outcomes in social care. We made use of previous studies to review the domains and levels, both empirically and in the light of our conceptual thinking, which had moved on since the earlier measures had been developed. Although service users had been involved in previous work, early versions were based on the perspective of those observing (e.g. researchers), commissioning and providing social care. We consulted with service users through general advice (the LSE PSSRU user and carer advisory group), a review of the literature16 and in the cognitive testing stages of this work and linked studies. 46

In this chapter, we describe the reviews of the previous measures and our conclusions in terms of the domains, levels and wording requirements needed for the development of the individual items for our instrument. We then describe the results of the cognitive testing of those new items taken through to the next validation stage.

Previous development and review of Adult Social Care Outcome Toolkit

The ASCOT measure was originally based on the Older People’s Utility Scale (OPUS), a preference-weighted measure of social care outcomes for older people. 5 A Delphi exercise, asking respondents to focus on the key domains of outcome for older people, identified five domains (food and drink, personal care, safety, social participation and involvement, and control over daily living). Drawing on the literature and expert opinion, the OPUS measure was later extended to adults aged < 65 years, and included the domains occupation, accommodation cleanliness and comfort, caring for others and living in own home. 47 These reflected both the concerns of social care for younger age groups, such as parenting, and the objective of moving beyond ‘key’ domains to a more comprehensive measure covering all aspects of social care. A single question was used to reflect each domain, with the exception of living at home, which was simply recorded, reflecting whether the person was living in their own home or in a care home setting. The wording of the individual items was cognitively tested and changed as part of development work for a survey of younger adults. 46 A pilot preference study was conducted, providing provisional preference weights anchored to a financial attribute using discrete choice experiments (DCEs). 10 This version of the measure has been used in a number of studies,4,46,48 including a major evaluation of the policy of individual budgets, a new form of consumer-directed support, in England. 49

As part of the Outcomes of Social Care for Adults (OSCA) review of the conceptual basis for the measure, we noted that the ongoing and intimate nature of most social care interventions meant that care itself became a part of people’s lives, which meant that the process of care-giving is important for people’s QoL. The literature review found 15 studies in which aspects of process were identified as important to people’s QoL, including such factors as being valued and treated with dignity and respect,49–54 being listened to/having a say,49–51 having kind staff with whom one can have good relationships55–58 and receiving a service that is both responsive and reliable. 59–61 For the most part, we would expect these aspects of process to be reflected in the SCRQoL domains. For example, people report that the reliability of a service affects their sense of control62 and may have consequences for having needs met in other domains. We would expect the impact of abuse and neglect to be picked up primarily in people’s sense of personal safety. However, some effects of the process of delivering care were unlikely to be captured, in particular those that have an impact on the psychological well-being of service users or their sense of personal dignity. We therefore decided to include in our measure an additional dignity domain, which captured the effect of the process of care-giving on a service user’s sense of self-esteem. Appendix 1 shows a table summarising the development of the domains through the different studies.

Conceptual work undertaken as part of the MOPSU study14 (Measuring Outcomes for Public Service Users) had identified the importance of distinguishing between reflecting Sen’s ‘capabilities’ and ‘functionings’. 31,32 Wording of early versions of the measure had tended to reflect ‘functioning’ states most closely. One version of ASCOT developed for low-level interventions had focused on reflecting capabilities. 9 We wanted our measure to reflect the full range from the very fundamental level where functioning levels are so low they could lead to mental and physical health implications, through to ‘capability’ states, in which people have real choice and their circumstances can be seen to reflect the objectives of the ‘personalisation’ agenda in social care. 11,40

Analysis of previous versions of Adult Social Care Outcome Toolkit

At the time of the review for this study, the instrument had most recently been used in two studies: (1) a postal UES in 2007 of 2228 people with physical and sensory impairments (PWPSI) aged 18–64 years receiving help to enable them to live in their own homes and (2) in an evaluation of the use of individual budgets in social care, in which interviews were conducted with 959 people from all service user groups (the study evaluating the use of individual budgets – IBSEN). The data sets from these studies were analysed to explore empirically, first, whether there were any problems with the items as phrased and, second, the choice of domains. The data sets and analysis are described in detail in Appendix 2. Here we summarise the results and implications for the development of the measure.

In both data sets the domains included:

• control over daily life

• personal cleanliness and comfort

• food and nutrition

• accommodation

• safety

• social participation

• occupation.

In addition, the data sets included an item on ‘caring for others’, but we did not include this in our analyses as it was not seen as relevant by a very high a proportion of service users. In the IBSEN study, each item had three levels, reflecting:

• No needs Where any needs experienced are minor or the type of variation in experience people might normally expect in the absence of impairment.

• Low or some needs Where there are needs that lead to lower levels of well-being, but that are not likely to have any long-term consequences if unmet.

• High needs Where the needs are of such a level of severity or number that mental or physical health problems are likely to ensue in the short or longer term.

In the UES self-completion version, an additional level was included that was intended to distinguish whether ‘no needs’ was due to those needs already being met by services.

The response rates for the items in both data sets were generally good. However, the number of missing data was higher for the occupation and social participation items, suggesting that these could be improved. Response rates also varied across client groups; they were particularly good for PWPSI, and poor for people with learning difficulties (PWLD) and people with mental health problems (PWMH). It is possible that this pattern arose because the phrasing of the items used in both studies was developed in cognitive interviews with PWPSI. 46

Across both of the data sets, the majority or the largest proportion of respondents chose the ‘no needs’ response option. [It should be noted that the distribution of some of the items (control over daily life, safety and food and nutrition) was better in the IBSEN than in the UES sample, which could be explained by the lack of the second ‘service help’ level in the IBSEN data set. This level could have been ‘taking’ respondents from the lower response options as well as the ‘no needs’ response option.] In addition, for the accommodation cleanliness and comfort, food and nutrition and personal care items, very few people in either data set chose the lowest response option. On both counts, this situation is desirable, as it implies that services are successful in compensating people for their impairments. However, it also means that the distribution of scores for any scale composed of these items is likely to be highly skewed, with most people to be found towards the high SCRQoL end of the scale and very few people towards the low SCRQoL end. The concentration of respondents at the high end of the measure could make the measure insensitive among populations with high SCRQoL. As we assumed that we would be unlikely to observe many people in poor SCRQoL situations, given the existence of services and the availability of help from other sources (family, friends and privately paid-for help), it is important to have high sensitivity at the high SCRQoL end of the scale.

In both data sets a similar relationship was observed between pairs of items. The social participation and occupation items had strong correlations, as did the personal cleanliness and comfort item with both food and nutrition and accommodation cleanliness and comfort. The strong correlation may indicate some conceptual overlap in these domains, which might cause problems in the preference elicitation study. In the case of the personal cleanliness and comfort, food and nutrition and accommodation cleanliness and comfort items, this explanation seemed unlikely, as the wording of the domains and the ideas the domains represent do seem conceptually distinct. A more likely explanation of the strong correlations noted is that they are all areas of life for which poor outcomes would have severe health implications, and service users (and service providers) are likely to express a preference towards having needs in these areas met. In other words, the strong correlation is a result of the similarity in users’ preferences and the response of services to needs in these areas. There is more similarity in the concepts expressed by the social participation and occupation domains, and it was possible that there was some conceptual overlap between them. However, these two areas tend not to be priorities for social care services, so it may have been this perspective that was causing the strong correlation between these two items. Factor analysis for both data sets seemed to indicate that the items formed a weak unidimensional scale, with many of the items having high unique variances, suggesting that the items were conceptually distinct.

Development of the new Adult Social Care Outcome Toolkit questions

The findings from the theoretical and literature review of the previous ASCOT instrument identified the need to develop a new dignity domain, and to ensure that Sen’s31 concept of capability was incorporated in the measure. From the empirical analysis we concluded there was a need to develop greater sensitivity at the high SCRQoL end of the scale by attempting to split the ‘no needs’ response option into two options: one representing the ideal situation, the other in which there are no real needs but the person considers the situation to be just adequate rather than completely satisfactory. In addition, the empirical work identified the need to improve the wording of the social participation and occupation domains, as well as to improve all the domains so that they were more applicable to older people, PWLD and PWMH. In summary, therefore, the literature and empirical review of the previous version of the measure suggested developing a version of ASCOT composed of nine domains, summarised in Table 2, where each question (excluding living in own home) has four responses. The meaning of these response options is summarised in Box 1.

In developing a new set of questions, we set out what each of the response options would look like for each of the domains. This is summarised in Appendix 3. This activity helped us to think about appropriate wording, and ensured that the whole team was clear about what each domain meant and what situation each response option was intended to represent. Where possible, we used terms such as ‘want’ or ‘like’ and ‘adequate’ to distinguish between the preferred situation and the ‘needs met’ or ‘mustn’t grumble’ state, the aim being to reflect capability differences. Need states were described using terms that reflected amount or frequency of events, closer to functioning states. Once the items had been developed, we assessed them using the questionnaire appraisal system63 before testing them with service users. We explain how the questions were tested, and present the results of this testing in the next section.

Testing the revised Adult Social Care Outcome Toolkit instrument

Conclusions

The OSCA project built on previous versions of the ASCOT measure, drawing on conceptual developments, the literature, consultations with stakeholders and empirical evidence to cognitively test a revised measure with service users. This measure included a new domain (dignity) and a fourth level for all of the domains intended to increase sensitivity and reflect capabilities. The cognitive testing suggested that in taking the dignity domain forward it would be helpful to include an additional question that allowed respondents to reflect their general attitude to receiving any help, as some people have problems coming to terms with needing help at all. The living in own home domain was dropped as it appeared to be double-counting other domains, rather than reflecting an additional care setting or location dimension to our measure. The next step was to explore the validity of the revised measure.

Introduction

Validity assesses the extent to which an instrument measures what it is intended to represent. In Messick’s words,69 it is ‘an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores or other modes of assessment’. The ASCOT measure is intended to be of use in economic evaluations and to provide information for decisions about resource allocation across social care. It should enable decision-makers to compare the value of different types of social care provision, such as a meals service with a home care worker. Validating ASCOT is therefore about identifying the extent to which the instrument captures the value of social care. 70

The ASCOT consists of several components to capture the value of social care. The main instrument is composed of two types of questions. The first set of questions asks people to rate their current SCRQoL state in terms of eight domains or attributes. A second set of questions then requires people to rate their current SCRQoL state in the absence of the ‘intervention’, within seven of the eight attributes, where ‘intervention’ can be defined variously according to the purpose of the study. The dignity attribute does not have an item for the second set of questions because it is process based and it cannot be asked when people are not receiving services. We refer to the second set of questions as the ‘expected’ SCRQoL items and the first set as the ‘current’ SCRQoL items. The third component of ASCOT is a set of preference weights that can be used to attach a value to each SCRQoL state. (The generation of preference weights is described in Chapters 4 and 5.)

The different sets of questions serve different purposes and can be used in a variety of contexts. The current items capture the prevailing SCRQoL state of the individual and could be used to compare the states of otherwise equivalent groups (such as matched samples). They could also be used for evaluation of interventions, where they could be administered before and after an intervention, to generate pre- and post-test scores, where the pre-test scores act as a proxy for the expected SCRQoL state in the absence of the intervention and the difference in the estimated effect. The alternative, pragmatic approach proposed in Chapter 1 is to ask individuals what their expected SCRQoL in the absence of services is directly, with the difference between that and currently experienced SCRQoL representing the contribution that social care makes to a person’s SCRQoL. For the expected score, dignity is assumed to be at the second level – where the care process has no impact on the person’s sense of self-worth. All of the SCRQoL measures (current, expected and gain) can be preference weighted.

The psychometric criteria of ‘construct under-representation’ (the failure to capture important aspects of the concept being measured) and ‘construct-irrelevant variation’ (when responses to the measure are influenced by factors irrelevant to the concept being measured) are useful for thinking about validity in the context of valuing social care. 71,72 However, as Brazier et al. 70 recognise, the psychometric approaches used to determine validity need modification to make them applicable to a preference measure. Brazier et al. 70 identify three aspects of preference measures that require validation: the descriptive system, the valuations or utility weights, and the empirical validity of the instrument, which refers to whether people, through their behaviour in practice, appear to value the different states in the way that they are valued in the measure.

Here we focus our assessment of validity on the validity of the descriptive system, which refers to the choice of domains, the specification of the items in the instrument, and the ability of the instrument to detect changes or known differences in SCRQoL. This was achieved in four separate sets of analysis. We first examined the construct validity of the individual items in terms of whether or not they reflect the concepts as intended. We then evaluated the construct validity of the three preference-weighted scales by exploring their ability to detect known differences in SCRQoL. In this we wanted to establish:

• current SCRQoL scale as a measure of social care-related QoL

• expected SCRQoL scale as a measure of social care need

• gain in SCRQoL scale as a measure of the contribution of services to SCRQoL.

Methods

Description of sample

A total of 566 contacts were attempted from a sample of 778 respondents to the older home care UES who were willing to participate in further research; this produced 301 (53%) complete interviews. A few refused (n = 18, 3%) or had died (n = 4, 1%), but most non-participants were not contactable (n = 243, 43%). The characteristics of the sample are shown in Table 3. As would be expected among publicly funded older home care service users, the majority of the sample was female, > 80 years old, single and living alone. 13,82 However, compared with the population of publicly funded older home care users from which this sample was drawn, the sample had slightly fewer females (68% compared with 72%), slightly more people from a white ethnic background (98% compared with 93%) and a smaller proportion of people > 80 years old (60% compared with 68%). 13

Current social care-related quality of life

Summary score

Each domain was scored ‘0’ (high needs) through to ‘3’ (ideal state) and the scores for each domain were summed for each individual to produce a single score of current SCRQoL, taking values from 0 to 24. The mean score for this sample was 18.2 [standard deviation (SD) = 3.49, maximum (max) = 24, minimum (min) = 8]. The domain scores were also preference weighted using the provisional valuations from the MOPSU preference elicitation study. 10 The SCRQoL score based on this could take values from 8.13 to 35.97. The mean score for this sample was 30.3 (SD = 4.1, max = 35.97, min = 19.04) (Figure 1). The distribution of the two scores was similar and there were very few, very minor differences between the results reported here for the equal- and preference-weighted measures, so we have reported only results for the preference-weighted scores.

FIGURE 1.

Kernal density plot for distribution of scores to current SCRQoL measure. Kernel = Epanechnikov; bandwidth = 1.1678.

Table 6 shows that current SCRQoL was significantly correlated with the measures of well-being, control and autonomy, and HRQoL. The relationship with the CASP-12 control and autonomy subscale and the GHQ-12 was strong, with correlation coefficients being > 0.5 and close to 0.6. 85 As expected, there was a significant relationship with our indicator of HRQoL, EQ-5D, but less strong.

TABLE 6 - Correlation between preference-weighted current SCRQoL scales and other measures of aspects of QoL (n = 301)

*Significant at 10% level; **significant at 5% level; ***significant at 1% level.

Measures	Mean score (SD)	Correlation with SCRQoL
EQ-5D	0.29 (0.36)	0.41***
GHQ-12	13.14 (6.50)	–0.58***
Control and autonomy subscale of CASP-12	9.72 (3.67)	0.58***

Expected social care-related quality of life

The distribution of responses to the expected SCRQoL items is shown in Table 7. A small proportion of responses are missing for each item – between 1% and 4% of responses. The largest proportion missing is for the safety attribute, which implies that this is the hardest attribute for people to judge what their outcome state would be like in the absence of services. However, the missing responses were small for all items, indicating that this method is feasible with the population.

TABLE 7 - Distribution of responses to the expected SCRQoL items

Attribute	Frequency	%
Control over daily life
I would have as much control over my daily life as I want	29	9.8
I would have adequate control over my daily life	41	13.9
I would have some control over my daily life	123	41.6
I would have no control over my daily life	103	34.8
Total	296	100.0
Personal cleanliness and comfort
I would feel clean and able to present myself the way I like	85	28.6
I would feel adequately clean and presentable	62	20.9
I would feel less than adequately clean or presentable	71	23.9
I would not feel at all clean or presentable	79	26.6
Total	297	100.0
Food and drink
I would get all the food and drink I like when I want	160	54.1
I would get food and drink adequate for my needs	71	24.0
I wouldn’t get all the food and drink I need, but no health risk	18	6.1
I wouldn’t get all the food and drink I need, and health risk	47	15.9
Total	296	100.0
Accommodation cleanliness and comfort
My home would be as clean and comfortable as I want	144	48.3
My home would be adequately clean and comfortable	84	28.2
My home would be less than adequately clean and comfortable	39	13.1
My home wouldn’t be at all clean and comfortable	31	10.4
Total	298	100.0
Personal safety
I would feel as safe as I want	100	34.6
I would feel adequately safe, but not as safe as I’d like	73	25.3
I would feel less than adequately safe	61	21.1
I wouldn’t feel at all safe	55	19.0
Total	289	100.0
Social participation
I would have as much social contact as I want with people I like	81	27.6
I would have adequate social contact	82	27.9
I would have some social contact with people, but not enough	85	28.9
I would have little social contact with people and feel socially isolated	46	15.7
Total	294	100.0
Occupation
I would be able to spend my time as I want, doing things I value or enjoy	77	25.9
I would be able to do enough of the things I value or enjoy with my time	62	20.9
I would do some of the things I value/enjoy with my time, but not enough	116	39.1
I wouldn’t do anything I value or enjoy with my time	42	14.1
Total	297	100.0

Unlike the responses to the current SCRQoL items shown in Table 4, where the effect of services is captured in the SCRQoL state, in this measure of expected SCRQoL responses are much more evenly spread across all the response options, reflecting the fact that for this sample their SCRQoL would be much poorer in the absence of services. Only the food and drink attribute had over half the sample in the top ‘no needs’ level, although the accommodation cleanliness and comfort attribute also approached 5%, again suggesting that the wording was not working as we might have hoped.

As with current SCRQoL, the expected SCRQoL values can be summed to produce an equally weighted score of expected SCRQOL, which can take values from 0 to 24. The mean score for this sample is 13.8 (SD = 4.40, max = 23, min = 2, n = 281). Using the valuations from the MOPSU preference elicitation study,10 the mean preference-weighted score for this sample is 24.5 (SD = 5.6, max = 34.97, min = 9.85, n = 281). Figure 2 shows the distribution of the preference-weighted scores. Both this and the equally weighted distribution were close to normal.

FIGURE 2.

Kernal density plot for distribution of scores to expected SCRQoL measure. Kernel = Epanechnikov; bandwidth = 1.6419.

Activities of daily living were measured using a summary score of nine items: ability to manage stairs, go outdoors and walk down the road, get around indoors, get in and out of bed, use WC or toilet, wash face and hands, bath or shower or wash all over, get dressed and undressed, and feed oneself. Each ADL is scored on a scale from 0 to 3, where ‘0’ = ‘cannot manage at all’, ‘1’ = ‘manage with help from someone else’, ‘2’ = ‘manage on own with difficulty’, and ‘3’ = ‘manage on own’. The correlation between expected SCRQoL and a summary score based on these nine ADLs was highly significant (r = 0.36, n = 244, p < 0.001). This was higher than the correlation between current SCRQoL and ADLs (r = 0.29, n = 258, p < 0.001), but not by much. Such measures are expected to be associated with current SCRQoL, as it is harder for people to achieve good SCRQoL because of the increasing difficulty associated with compensating people for the effects of severe impairments.

Social care-related quality of life in the absence of services is difficult to explain using only an indicator of functional ability, because of the complexity of need and the role of forms of support and context (such as home design) that may also contribute to SCRQoL in the absence of services. The primary objective in measuring expected SCRQoL is to provide a pragmatic approach to assessing the impact of social care interventions, by comparing this with people’s current experiences, to measure SCRQoL gain.

Social care-related quality-of-life gain

We can illustrate the impact of services on different domains using a cobweb chart. Figure 3 shows this for this sample of older home care service users, with the darker-shaded area representing the impact of home care user service packages on SCRQoL. This shows that the contribution of services received by this group was greatest for personal cleanliness and comfort then control over daily life and smallest for occupation and social participation.

FIGURE 3.

The contribution of services to SCRQoL for each attribute.

For each individual, the welfare gain (or loss) for each attribute can be summed together to produce a single score of impact of services on SCRQoL (SCRQoL gain), which can take values from –24 to 24, where negative values indicate a loss of welfare and positive values indicate a gain. (The value for expected dignity is given by assuming that in the absence of services ‘The way I’m helped and treated does not affect the way I think/feel about myself’ is chosen.) The mean score for this sample was 4.42 (SD = 3.77, max = 22, min = –3). The two preference-weighted versions of this measure can also be subtracted from each other to generate a preference-weighted version of the gain in SCRQoL, which can take values from –35.97 to 35.97. The mean score for this sample is 5.74 (SD = 4.85, max = 26.12, min = –2.03, n = 281). The distribution of preference-weighted measure is skewed (Figure 4). The positive value for the sample means that, on average, this sample gained in SCRQoL as a result of social care. The distribution for the preference-weighted version was closer to normal.

FIGURE 4.

Kernal density plot for distribution of scores to gain in SCRQoL measure. Kernel = Epanechnikov; bandwidth = 1.3933.

The majority of the sample was using publicly funded home care services. A straightforward measure of service intensity is the number of hours of publicly funded home care received. However, 54 people in the sample reported not receiving publicly funded home care and, of those receiving publicly funded home care, a number also reported receiving other publicly funded services. Ideally, a measure of service intensity would capture these other sources of support. Therefore, we have constructed a second measure of service intensity: the cost of the care package, based on weekly levels of home care, day care and meals receipt, and unit costs of these for each council in our survey. 13

Table 8 shows the correlation coefficient for the relationship between both of the service intensity variables and the gain in SCRQoL and current SCRQoL scales. We have also included the correlation between the log of service intensity and both versions of SCRQoL to account for the fact that the relationship is likely to be non-linear.

Interestingly, and consistent with research elsewhere using different measures of social care outcome, the SCRQoL current measure has a non-significant and, in many cases, a slightly negative correlation with service intensity. 86,87 We would expect people with greater needs generally to receive larger packages of care. No significant correlation would indicate that a similar SCRQoL is being achieved irrespective of need, suggesting that if services are being allocated rationally they are compensating as we would expect. Thus, current SCRQoL captures the result of care and other factors that influence SCRQoL.

The significant positive correlation between service intensity and gain in SCRQoL indicates that the more intense the service, the greater the gain in SCRQoL. There was some evidence that the relationship between service intensity and SCRQoL gain is non-linear, where service intensity is measured using hours of home care. Of course, in addition to the factors that we identified above as likely to be associated with expected SCRQoL, there will be other factors such as the quality of care provided. Multivariate analyses are required to establish the underlying relationship between service intensity and impact. Nevertheless, we have the expected relationship between service receipt and SCRQoL, suggesting that the gain in SCRQoL measure is performing as expected.

Conclusion

The aim of the stages of work in the development of the measure reported in this chapter was to ensure that the measure to be preference weighted was practical and had good psychometric properties. The analyses conducted provide some evidence for the validity of ASCOT and the feasibility of using the measure with social care service users. Response rates were good for all of the items in both sets of questions, which implies that both the current SCRQoL items and the hypothetical pre-test approach used here are feasible with this population.

The detailed investigation into the validity of the measures also provided some evidence for validity. There was good evidence for the validity of the descriptive system for ASCOT: each attribute seemed to capture a different aspect of SCRQoL, and each attribute had the anticipated relationship with other variables capturing similar and dissimilar concepts. There was also some evidence for the validity of the current and impact SCRQoL scales, which appeared to operate as intended.

Concern about the distributions of both the current and expected accommodation and food and drink items was compounded by a lack of differentiation in preferences for those items in parallel work that was undertaken on establishing preference weights for this version of the measure as part of the MOPSU project (see Appendix 6). This suggested that respondents did not see the ‘capability’ difference we were attempting to distinguish. Moreover, such weights combined with the distributions meant the items would not add substantially to the measure. We revisited the wording and took opportunities through parallel studies to test out the revised items with service users. 8 The final version of the instrument is shown in Appendix 5, and population and equipment service user distributions of the ‘current’ items using this are shown in Chapter 6.

Introduction

As described in Chapter 1, the first phase of the study included an investigation into alternative approaches for modelling preferences of the general population for social care outcomes, and an exploration of the applicability of these approaches to a service user population. The aim was to support the design of a large-scale population and service user preference study to develop weighted measures of social care. In addition, we wanted to explore the feasibility of anchoring the measure to the state of ‘being dead’, to provide the potential for using the measure as a social care equivalent of the widely used QALY in health care. The work was undertaken in parallel to the development work on the measure itself, so used early versions of the domains or attributes.

In this chapter we start with a brief consideration of the potential options open to us, before describing the results of the pilot feasibility study that investigated the use of DCE and BWS methods. We then describe the results of cognitive testing of these techniques with service users and investigation of the TTO method in the context of social care.

Eliciting preferences

In eliciting preferences, there are a number of considerations that feed into the validity and reliability of the resulting preference weights. These include ensuring that the task is not too complex, so we can have confidence in the data collected from respondents. This is particularly an issue with our measure, which has a relatively large number of domains or attributes. Ideally, we also want to minimise the assumptions we make about what people are doing in expressing their preferences, in particular so that their responses can be interpreted on a cardinal basis. We want the process used to be consistent with expressions and interpretation of preferences in economic theory. In addition, ideally we want it to be possible to anchor the weights to some other characteristic (such as the ‘being dead’ state) or widely understood unit (such as money) in a way that provides potential for a wider use than a free-floating index that can only be compared with itself.

Typically, public preferences are elicited in nationally representative valuation exercises using the visual analogue (VA), standard gamble (SG) or TTO. 68 These tools require respondents to manipulate probabilities or lengths of life and so rely on an assumption of cardinality in responses. Theoretical and empirical problems with these methods88 have been accompanied by interest in tasks that require only ordinality in responses, such as DCE and BWS.

Unlike the related field of conjoint analysis (CA), DCE and BWS are based on a sound, well-tested theory of decision-making and choice behaviour known as random utility theory (RUT). 88–91 CA is used to establish the relative importance of different attributes in the provision of a good or service, as well as to estimate the overall utility of a given commodity, with levels of given attributes. Individuals’ preferences are recorded as rankings or ratings, both of which are cognitively demanding tasks. The models developed (analysis of variance, ordered probit or ordered logit) are purely statistical, with no behavioural basis. Traditional CA investigates the process by which people evaluate one option at a time and state a preference for each option, which contradicts with most real-life decisions that involve comparisons of alternatives. 92 Experimental design techniques are used to construct a sample of scenarios – presented as combinations of attribute levels – from a full factorial set. In a DCE, respondents are asked to choose an option (scenario) out of a set of two or more options, which ultimately involves a comparison of indirect utility functions. Analysis of data is conducted using discrete choice models,92 and, provided that any variance heterogeneity on the latent utility scale is properly accounted for,93,94 the estimated coefficients of each of the attributes in the model are unbiased utility weights that represent the relative importance of domain levels.

Discrete choice experiments have been used extensively to facilitate analyses in the fields of transport and environmental policy. However, they can also be used to develop preference weights for different instruments, and work is under way to do so for the EQ-5D-5L (EQ-5D 5-level) for measuring health outcomes (as a supplement to a TTO valuation)95 and the ICECAP capability indices. 73,96 They have also been used previously to explore the potential for developing preference-based outcome measures in social care,97,98 including addressing the problem of a large number of attributes by using two overlapping experiments. 68

The BWS method is an alternative preference elicitation method that also requires only an assumption of ordinality. It was developed by Louviere and Woodworth99 and its first application was published in 1992100 illustrating Case 1 (the ‘object’ case). The method gained popularity in health and social care when the properties of Case 2 (the ‘profile’ – previously called ‘attribute’ case) were proved,101 and a guide to its use was published. 102 Flynn103 provides an overview and theoretical discussion of the different cases of BWS. Case 2 has particular advantages in valuation studies that seek to elicit general population preferences for important attributes of QoL (or whatever maximand is of relevance to policy-makers104). In particular, it presents profiles one at a time, rather than in choice sets of size two or more as in a traditional DCE. This is important when respondents do not have experience of making choices in the particular area of application: keeping two or more profiles in mind at once is likely to be a harder task, leading to an increase the size of the random utility component and reduction of the statistical efficiency of the preference elicitation.

Best–worst scaling is a relatively new method, and there is a lack of empirical evidence that demonstrates the superiority of BWS over DCE in terms of practical considerations in the field (e.g. respondents’ fatigue) and the robustness of the modelling estimates. As shown in Table 9, DCE and BWS are promising approaches as they satisfy a number of theoretical and practical considerations. However, with such approaches there is an issue of anchoring. Anchoring refers to the adjustment of utility weights obtained from DCE or BWS experiments on to a ‘0–1’ scale, where ‘0’ represents ‘being dead’ and ‘1’ is the social care equivalent of ‘perfect health’, respectively. Anchoring also allows integration of information about the QoL and the length of time over which that QoL is maintained, and wider comparisons across interventions. However, this presents methodological challenges, both in terms of how to get respondents to consider a ‘being dead’ situation in the context of choices regarding the outcomes of social care, and how respondents interpret whether any given intervention (and resulting life state) is likely to increase or reduce the time until death. Moreover, including ‘being dead’ as a state (like the social care states) will neither conceptually nor empirically estimate the trade-off between length of life and QoL. 104

One approach to the problem is to undertake a complementary piece of work to allow us to anchor the measure to the ‘being dead’ state, while using the main preference study to estimate the individual preference weights. We identified TTO, seen as a simpler alternative to SG (as shown in Dolan et al. 75) as a method that would allow us to explicitly identify the relationship between population preferences for SCRQoL states and ‘being dead’, thus, providing an anchor for the measure. TTO was chosen in preference the other widely used cardinal technique of visual analogue scaling (VAS). TTO asks the respondent to explicitly consider a trade-off between length of life and QoL that has been argued to have a stronger grounding in theory than a simple rating task. 103

The main tasks of the feasibility stage of this aspect of the work were, therefore, to compare the discrete choice and profile-case BWS to elicit values for different dimensions of SCRQoL. This involved evaluating both the extent to which valuations of SCRQoL states obtained through the two approaches produced similar results, and the relative complexity of the tasks, particularly from the perspective of service users. In addition, we needed to test how acceptable TTO methods were when applied to SCRQoL states, as this is not the type of trade-off that might more realistically be considered with respect to health states. Because of the potential sensitivity of the issue, it was decided to limit this aspect of the study to members of the general population.

Comparison of best–worst scaling and discrete choice experiment valuations

In order to determine the extent to which valuations of SCRQoL states obtained through a BWS were comparable to those obtained through DCE, we undertook a pilot study with 300 members of the general population. Nine domains were in the measure when we were designing the pilot study: food and drink, personal cleanliness, accommodation, safety, social participation, occupation, control, dignity and living in own home (later dropped – see Chapter 2). Each domain had four levels, except for living in own home, which had two levels (living in own home, not living in own home). For purposes of clarity, and in order to avoid wording that may lead to some domains dominating the choices, the dignity domain was described to respondents as ‘the way I am helped’ and the employment and occupation domain was presented as ‘use of my time’.

Discrete choice experiments

The task of trading-off across nine different QoL domains in a single DCE would be particularly challenging. We drew on previous work that had used the approach of splitting domains across two DCEs (DCE1 and DCE2), with overlap in some domains to allow models from the two experiments to be based on a common utility scale. This study had demonstrated that the approach produced consistent values, and the utility parameters of overlapping domains were equal. 10

The principles of grouping the domains into the two experiments were the following:

• Control over daily life and dignity should appear in both experiments.

• Safety, personal care and food and drink should appear in the same experiment, as these could be seen as core or basic outcomes of social care services.

• Employment and occupation and social participation and involvement should also appear in the same experiment.

The logic of this allocation of domains between the two experiments was, first, that people may make trade-offs between personal care and safety. An example from earlier cognitive interviews was a lady who would prefer to be clean and put herself at some risk getting into the bath, rather than be safe and unclean. The other reason is because safety, personal care and food and drink can be represented as the core outcomes of social services. Cleaning the house, social participation and being active/occupied can be seen as at a less fundamental level in terms of such scales as Maslov’s hierarchy of needs. 37 By splitting the attributes across two experiments, we would not be able to explore any interactions in preferences for domains that were in different experiments, so it was necessary to make a priori assumptions about which interactions were likely to be most important when grouping the domains. The final allocation of the domains between the two choice experiments is shown in Table 10.

TABLE 10 - Grouping of domains between the two choice experiments

DCE1		DCE2
1	Food and drink	4	Accommodation cleanliness and comfort
2	Personal care	5	Social participation and involvement
3	Safety	6	Employment and occupation (use of my time)
7	Control over daily life	7	Control over daily life
8	The way I am helped (dignity)	8	Dignity (the way I am helped)
		9	Living in own home

Both DCE1 and DCE2 used a forced D-efficient choice design and each of the design matrices included 128 situations. These matrices covered sufficient domain-level combinations to allow all two-way interactions between domain levels in the same experiment to be estimated. Obviously, 128 situations for each experiment would be too difficult for one respondent to evaluate. Therefore, the design matrices were divided into 16 orthogonal blocks, so that each respondent was presented with eight situations per choice experiment. Each choice scenario presents two situations as a list of domains at different levels. Examples of the introductory scripts and choice exercises in DCE1 and DCE2 are shown in Box 3 and Figure 5, and Box 4 and Figure 6, respectively.

BOX 3 - Script introducing DCE1a

From this point on I would like you to imagine that you are in a situation where, through either illness or accident, you are no longer able to do everything you might expect to do for yourself without some assistance

I will ask you to look at a series of scenarios – in each we will present two different situations to you. These situations may have some good and bad aspects

In the following choices we will be concerned with these aspects of quality of life:

• food and drink

• personal care

• safety

• control over daily life

• the way I am helped

I would like you to weigh up the pros and cons of finding yourself in each situation. Then I would like you to tell me which situation you would consider to be the worst

These situations are imaginary, but I would like you to think about how you would feel if you were in these situations. There are no right or wrong answers to these choices; we are only interested in your views

Scripts for DCE1 and DCE2 varied according to the order of appearance, e.g. DCE1 first and DCE2 second. Boxes 3 and 4 assume that DCE1 and DCE2 are presented first during the survey.

BOX 4 - Script introducing DCE2

From this point on I would like you to imagine that you are in a situation where, through either illness or accident, you are no longer able to do everything you might expect to do for yourself without some assistance

I will ask you to look at a series of scenarios – in each we will present two different situations to you. These situations may have some good and bad aspects

In the following choices we will be concerned with these aspects of quality of life:

• accommodation cleanliness and comfort

• social participation and involvement

• use of my time

• control over daily life

• the way I am helped

• living in own home

I would like you to weigh up the pros and cons of finding yourself in each situation. Then I would like you to tell me which situation you would consider to be the worst

These situations are imaginary, but I would like you to think about how you would feel if you were in these situations. There are no right or wrong answers to these choices; we are only interested in your views

FIGURE 5.

A choice exercise example from DCE1.

FIGURE 6.

A choice exercise example from DCE2.

Best–worst scaling experiment

The BWS experiment used exactly the same attributes as the DCE but, rather than splitting them into two groups, all nine domains were presented in a single situation. Previous work had found that as many as nine domains in each situation is feasible as respondents are asked only to decide which of the domain levels presented is best and which is worst. 97 Choice situations in the BWS experiment were specified using an orthogonal main effects plan (OMEP). The full plan consisted of 48 × 21 combinations, which were reduced to 32 situations in eight orthogonal blocks of four scenarios each. Each respondent was assigned a random sample of three blocks, so was asked to indicate his/her best and worst domain levels in 12 different situations. Figure 7 shows an example of a BWS exercise, where the respondent was asked to choose the best and then the worst domain.

FIGURE 7.

Best–worst scaling choice exercise.

In selecting the situations, it was important to consider the potential for bias resulting from differences in the size of the random utility component variance by attribute. 102 For example, presenting a situation in which one attribute was at its ‘top’ level (while all other attributes were at an intermediate level) was likely to make the ‘best’ choice easy (unless the levels of that attribute were all of moderate size and similar on the latent scale). Therefore, the random utility variance, ceteris paribus, would be small compared with a situation with all attributes at intermediate levels. As the OMEP was a small fraction of the full factorial, it was possible to avoid such problematic situations; the coding of domain levels was chosen to avoid designing situations defined by every attribute at its ‘top’ level, those with every attribute at its ‘bottom’ level and ‘easy to choose’ situations of the type described above.

All respondents participated in the DCE1, DCE2 and BWS experiments. The order of the experiments was randomised in order to be able to test whether or not the order of appearance of experiments and, hence, whether or not fatigue and boredom or self-learning effects (i.e. the respondent gains more experience as he/she completes more exercises) had any effect on the responses of participants. There were four ordering possibilities, shown in Table 11, which were randomly allocated across respondents. This allowed development of models that enable testing differences, for example between responses where BWS exercises were presented first and BWS exercises were presented last.

TABLE 11 - Order patterns of appearance of the experiments

Pattern	Order of appearance
	First	Second	Third
1	DCE1	DCE2	BWS
2	DCE2	DCE1	BWS
3	BWS	DCE1	DCE2
4	BWS	DCE2	DCE1

Testing for ordering effects

Prior to the development of the final models, we tested whether or not the order of appearance of the DCE1, DCE2 and BWS experiments (see Table 11) resulted in significant differences in the estimated models. We divided the sample in each model into three segments, depending on whether the experiment (i.e. DCE1, DCE2 or BWS) was completed first, second or third. For each segment we specified different scale parameters to reflect this ordering. In the case of BWS, only two scale parameters were specified reflecting whether the BWS was presented first or last.

As shown in Figure 8, the scale parameters of the first branch (i.e. DCE1 appears first) were fixed at the value of ‘1’ for identification. The objective is to estimate the scale parameters ϑ₂ and ϑ₃, after imposing the restriction that domain-level coefficients be equal across branches. If ϑ₂ and ϑ₃ are not significantly different from 1 then there is no evidence of a difference in the variance of the unexplained (unobserved) component of utility according to the experiment order. The same test can be applied for DCE2. As shown in Figure 9, the case of the BWS experiment requires only one scale parameter to be estimated, because the BWS appeared either first or last. Table 14 shows the values of the scale parameters in the DCE1, DCE2 and the BWS data, along with their corresponding standard error (SE) and t-statistic values. The results indicate that scale coefficients are statistically significant from 0, but are not significantly different from 1. As a result, the order of appearance does not affect the level of noise within the responses.

FIGURE 8.

Testing differences in coefficient estimates because of the order of appearance of DCE1. Similar is the case for DCE2.

FIGURE 9.

Testing differences in coefficient estimates because of the order of appearance of BWS experiment.

TABLE 14 - Scale coefficients for testing the influence of experiments’ order of completion

Wald test = value of parameter (ϑ – 1)/SE. If the computed absolute value is smaller than the critical value ± 1.96, then we cannot reject the hypothesis that the scaling parameters are statistically different to 1.

More detailed description of the econometric analysis and discussion of modelling results can be found in Potoglou et al. 104

	ϑ2				ϑ3
	Value	SE	t-statistic	Wald testa	Value	SE	t-statistic	Wald test
DCE1	1.14	0.11	10.0	1.27	1.11	0.15	7.6	0.73
DCE2	0.873	0.11	7.6	–0.12	1.22	0.17	7.1	1.29
BWS	1.04	0.04	28.9	1	–	–	–

Comparison of values between discrete choice experiment and best–worst scaling

The results from DCE1, DCE2 and the BWS experiments can be compared if we look at the marginal values of moving from the lowest level (e.g. Dignity_1, which is constrained to 0) to the highest level (e.g. Dignity_2 and Dignity_4 in Figure 10) in each domain.

FIGURE 10.

Rescaled common domain weights across BWS, DCE1 and DCE2.

The models have different scales, hence the coefficients cannot be directly compared,105 but we can look at the relative size of the differences by using one of the domain levels as a common denominator and scaling all others relative to this. In this case, we have chosen the highest level of need of the control domain (i.e. Control_4), which was strongly estimated in all models. (All negative domain level coefficients have been divided by the negative coefficient of control, and that is why the coefficients of each domain level in Figures 10–12 appear to be positive.) Figures 10–12 provide a comparison of the relative values and SEs (see Hess and Daly106 for computation of SEs). Figure 10 presents the comparison of domain coefficients that were common across DCE1, DCE2 and BWS. Figure 11 shows the comparison of domain coefficients between BWS and DCE1, and Figure 12 shows the coefficients that appeared in DCE2 against those estimated in BWS.

FIGURE 11.

Rescaled common domain weights across BWS and DCE1.

FIGURE 12.

Rescaled common domain weights across BWS and DCE2.

The results revealed a broadly similar pattern in preferences using BWS and DCE. In 8 out of 11 cases, the domain-level weights common between DCE1 and BWS were not statistically different at the 95% significance level. Only the third level of control and second and third level of food and drink were statistically different at the 95% and 99.9% confidence intervals (CIs), respectively. In DCE2 and BWS, 8 out of 12 weights were not statistically different from each other at the 95% significance level. The rescaled BWS estimates were statistically different to those from the DCE2 for employment and occupation (level 3), social participation (level 3) and living in own home.

Finally, the values placed on dignity and control between the two separate DCE exercises (for which these were the common attributes) were not statistical different. As a result, the two different groupings in the design appear to support consistent valuations of these common attributes across respondents.

Service user cognitive testing

The results of the pilot preference study suggested that while there might be some advantages for BWS over DCE, in that all of the attributes or domains could be considered together, the resulting weights would not be dependent on the technique used. However, if we are to have confidence in the estimates, we need to be clear about the basis on which people are responding. In particular, we wanted to explore the acceptability and cognitive burden of this unusual type of questioning among service users. We wanted to find out how best to present the tasks to people, whether or not people understood what was expected of them, and whether or not they could manage the tasks.

The testing of the approaches to preference elicitation was conducted at the same time as the cognitive testing of the basic measure using the same methods (see Chapter 2). We presented users with the tasks as they would be presented in the actual study, and asked them to complete them as they would if the interview was part of a real study designed to elicit preferences. When giving their choices, we asked the interviewees to explain and give reasons for the choices they made.

Time trade-off

The basic BWS and DCE models generate values on the latent scale for all social care states. The problem is how to anchor these on to the ‘QALY’ scale where ‘0’ represents ‘being dead’ and ‘1’ represents the social care equivalent of ‘perfect health’. One solution to this problem is to value the worst state described by a classification using a cardinal preference elicitation technique such as TTO, and use this to ‘anchor’ the BWS or DCE values on the QALY scale. This relies on using just one data point and has been shown to poorly predict mean TTO values for other health states. 107 A better approach would be to map from BWS to TTO using more data points and estimating the functional form.

The objective is to identify the point at which QoL states are viewed by respondents as equal to being dead – and to scale the utility scores accordingly. However, we needed to know whether or not such an approach would be acceptable and understandable for members of the general public in the context of social care rather than health-care outcomes. Moreover, we wanted to test using a computer-based format, rather than with a physical board and cards, which has been the predominant method used in the past.

The TTO exercise was designed following established protocols. 93,94 Respondents were asked to choose between having all of their social care needs met for a period of < 10 years, followed by immediate death (Life A); and having a lower QoL for 10 years (Life B). The point of trade-off is established by varying the duration of the ‘all needs met’ state (Life A), and asking respondents to indicate at which point they view Life A and Life B to be equal. An example of a scenario is shown in Figure 13.

FIGURE 13.

An example valuation scenario in the TTO experiment. Note: The upper state was subsequently replaced by ‘all needs met’.

Four waves of interviews were conducted. The first two waves asked 10 respondents to consider six SCRQoL states. These interviews helped to identify a number of useful changes, including:

• amendments to the introduction to the interview, to improve the presentation of the TTO tasks

• an increase from six to eight ‘evaluation states’ for the third and fourth waves of the testing, given that participants did not appear to be distressed, fatigue or bored when evaluating six states, and were likely to be able to evaluate more states

• minor revisions to the screen layout and choice instructions to assist in the clarity of the task

• a change in the format of Life A (best state), from a list of all domains at the ‘no needs’ level to a simple statement of ‘all needs met’, supported by a prior explanation of the levels associated with this ‘all needs met’ state in the introduction.

In total, 19 respondents were interviewed across the four waves of the cognitive testing. Slightly over half of the sample (11) was female, most (16) were white, and a range of age groups and socioeconomic groups were represented.

Data collected during the interviews were used to review general trends in the behaviour of individuals in these TTO choice tasks to examine whether their behaviour aligned with expectations and to identify issues which might require additional consideration. Exploratory analysis examined the relationship between the TTO results and the utility scores based on BWS. [These were based on the results of the MOPSU preference study10 (see Table 1).] In principle, we would expect that the TTO and utility scores would follow a monotonic relationship, i.e. the TTO score would increase as the calculated BWS utility score increased. In the majority of the cases, we were able to observe just such a monotonic relationship, although there were a number of outliers.

In the first two waves, three respondents valued mid-score utility states as being worse than being dead, suggesting that they either had extreme preferences or, more likely, had misunderstood the TTO tasks. This issue may have been resolved via the developing understanding of the interviewer, as only one respondent in the third and fourth waves responded in this way.

It was also notable that, in the third and fourth waves, three respondents had a uniform (and high) distribution of TTO scores, regardless of the state under evaluation. It may be that participants were unwilling to trade-off any time from their life – regardless of state – with being dead. Feedback from the cognitive interviews suggested that this was the case for at least one respondent, who refused to sacrifice any time in Life A to avoid Life B, and consistently chose Life B for 10 years.

Implications for the main stage design

To our knowledge, the pilot exercise we undertook was the first to test empirically the comparability of the profile-case BWS and DCE estimates. The results were encouraging. Overall, we observed similar patterns between the DCE and BWS, and in the majority of cases the estimated preference weights (when normalised/rescaled) were not statistically different. The differences that were observed in the BWS and DCE estimates might reflect bias in the latter caused by different respondents inferring different information about the attributes they did not ‘see’: omitting attributes from a DCE typically affects estimates of both the attribute level means and the variance (scale). 108 In BWS, respondents saw all of the attributes at once. The tests for preference equality suggested such effects at the aggregate level may be small. However, there may be important differences among subgroups. Any such differences would have implications for the generalisability of the results to the wider population. An important lesson for the main stage, therefore, was that the analysis should investigate – and, if necessary, control for – heterogeneity across different segments of the population (e.g. education level, place of residence).

In comparing the DCE and BWS approaches, BWS has the advantage that respondents are presented with all domains at the same time. BWS also provides more information than a traditional DCE for a given set of choices, as coefficients are estimated on a common scale (allowing the measurement of the relative value of the domains and not just differences within each domain). Moreover, BWS appeared to pose a smaller cognitive burden than a DCE in which respondents need to weigh up the combined value of all the attributes simultaneously. This was consistent with the findings in the wider literature (e.g. see Flynn et al. 102).

We felt it was important to investigate, and ideally reflect, the service user perspective in weighting our measure. Although the cognitive testing with service users identified some potential problems in establishing preferences, once the introduction and layout were clear we found that, overall, people were able to undertake the tasks. We established an approach for identifying people for whom the task was too cognitively demanding, and strategies for identifying and investigating the incidence and impact of strategies, such as taking into account only certain domains.

The results of the TTO testing were also encouraging. Respondents appeared to find the questions understandable and acceptable and, after some modifications, the computer-aided approach worked well. The lack of respondent fatigue suggested that we could increase the number of states presented from the minimum, which was desirable as there were potential non-linearities in respondents’ preferences across the range. 68 In addition, the exercise identified the importance of interviewer briefing and training for this type of interview, which helped inform the main stage processes.

As we described in Chapter 1, we took advantage of parallel studies to feed into this project. The preference study, conducted as part of the MOPSU research, is described in more detail elsewhere. 109 Key aspects from the perspective of the current study were that it allowed us to test the BWS approach, gathering further information about ‘second’-best and worst attributes in each scenario (see Chapter 5), with a revised, close-to-final version of the ASCOT measure, which excluded the living in own home domain with a general population sample. In addition to allowing us to extend the sample size for the main BWS analysis, this piece of work allowed us to generate weighted ASCOT states for the main stage TTO evaluation.

Introduction

Having finalised the design of the ASCOT instrument (see Chapter 3 and Appendix 5) and being informed by the results of the feasibility work on preference elicitation (see Chapter 4), we turn now to the main preference elicitation surveys and the development of utility weights for our measure. We start by describing the design of the BWS experiments, and the surveys and data on which we were able to draw for the main analysis. We present the analysis undertaken on the BWS data and the resulting domain weights for both the general population and the service user samples. We then turn to the design and results of the TTO experiments. In Chapter 6 we then provide a step-by-step example of how the utility weighting scheme can be applied by analysts to calculate a comparative score for different SCRQoL states, and ultimately a corresponding social care equivalent QALY score.

Design of the best–worst scaling experiment

The final version of the ASCOT measure has eight domains, each with four levels. As in the pilot study, the BWS tasks were developed using a fractional–factorial design, which reduced the full factorial plan (four8 tasks) to a matrix of 32 tasks. This design matrix was blocked using a procedure that sought to minimise the correlation of the levels being presented for the domains within the block (i.e. avoiding all respondents in one version of the survey always seeing a given domain at the same level). An additional consideration in defining the BWS tasks was the minimisation of ‘easy options’ or straightforward choices in each task. To achieve this, the domain levels were re-coded in the design matrices through an iterative procedure to ensure that each BWS task had more than one domain at the end point levels (i.e. level 1 or 4), thereby generating situations where non-obvious choices between domains were required. Each respondent was given eight BWS tasks.

Figure 14 shows an example of a BWS exercise used. In addition to choosing the best and worst domains, respondents were also asked to specify their second-best and second-worst choices. Specifically, the respondents indicated the best domain from the list of eight domains. The chosen domain was excluded from the list, and respondents were asked to choose the worst domain from the shortened list. The chosen domain was again excluded from the list and, following the same pattern, respondents were asked to indicate the second-best and second-worst domains of those remaining. Asking for second-best and second-worst choices reduces the effects of lexicographic and non-trading behaviour. Further, it is possible to gain additional information on the importance of the other issues in the absence of a domain that was first chosen as best or worst. 89,110 Finally, it is possible to draw more information per choice task as we obtain a partially complete ranking of four out of eight domains.

FIGURE 14.

Choice sequence in the BWS task.

The order of the attributes was varied for each respondent in order to prevent ordering bias and to be able to separate the effect of domain choice from the position of that domain within a task.

This BWS design was tested in the MOPSU study with members of the general population and it worked well. In that study, we used the version of the measure that was tested with older home care service users, which, as described in Chapter 3, was modified slightly as a result of the findings from both studies:

• accommodation cleanliness and comfort at level 3, and

• food and drink at levels 2, 3 and 4.

Although the MOPSU and OSCA surveys were conducted some time apart (in 2009 and 2010, respectively), we sought to maintain as much consistency as possible in their design so we could pool the data to provide a larger general population sample on which to base the final BWS-based utility weights.

Modelling the best–worst choice data

The analyses of the BWS general population and service user data were conducted using discrete choice models, refining initial models, and testing hypotheses and associations before settling on final models to use as an estimate for our preference weights:

• Estimation of SCRQoL domain weights after controlling for the order of domains in a given BWS task, sample level variance heterogeneity for best/worst responses, and excessive randomisation (in terms of the presentation of the tasks as a result of a fieldwork problem – described further in the following section) in the MOPSU data set.

• Estimation of domain weights after controlling for the order of domains in a given BWS task, variance heterogeneity for best/worst responses, excessive randomisation in the MOPSU data set, and with the additional consideration of variance heterogeneity occurring across different segments in the sample.

• Comparison of general population and service user domain weights based on the second set of models.

• Estimation of domain weights after controlling for the order of domains in a given BWS task, variance heterogeneity for best/worst responses, excessive randomisation in the MOPSU data set, variance heterogeneity occurring across different segments in the sample, and with the additional consideration of socioeconomic characteristics of respondents.

• Estimation of domain weights when the four-level domains used in this study collapsed into three levels by combining levels 1 and 2 in each domain and after controlling for the order of domains in a given BWS task, variance heterogeneity for best/worst responses, excessive randomisation in the MOPSU data set, and variance heterogeneity occurring across different segments in the sample (reported in Appendix 7).

Table 17 shows the number of observations available for modelling in the general population and the service users’ data sets.

TABLE 17 - Number of observations available for modelling in the general population and service users samples

Data	No. of choices per best–worst task (best, worst, second best, second worst)	No. of best–worst tasks per respondent	No. of respondents	No. of observations available for modelling
General population (OSCA and MOPSU)	4	8	929	29,728
Service users	4	8	367	11,744

Initially we developed multinomial logit models (MNLs)58,90 under the assumption that the best, worst, second-best and second-worst choices made by the same respondents were independent of each other (although from a diminishing set of choices). Also, the initial models were estimated without explicit consideration of the correlation between observations from the same individual. This model structure was considerably simpler to set up and had very short run times; thus, allowing a large number of different model specifications to be tested before moving to a more complex models. While this was a simplification with a potential for correlation within the responses from a given individual, both within a given BWS task and between BWS tasks, these correlations were accounted for in the final models, which are reported in the following sections.

Each domain that could have been chosen within the best–worst task was specified as a separate alternative within the choice model, and was given a utility function that took account of the level at which that domain was presented to the respondent in the task in question. The structure of this model is shown in Figure 15.

FIGURE 15.

Model structure in the analysis of the BWS data.

The MNL model was specified such that the utility function for a given domain was defined as:

• a linear-additive function of the products between the weights (coefficients) to be estimated and the dummy coding of the corresponding domain levels (only one of which will ever take a value of 1 for any given choice), and

• a number of dummy-coded variables to control for the position of a domain level in the best–worst task when that domain level was chosen as being best, worst, second best or second worst.

Some initial tests were undertaken to assess whether or not it was feasible to pool the data from the MOPSU and the main OSCA general population surveys, thereby providing the greatest return from the investment in data collection. Separate models were estimated from the data sets from the two waves, which both returned well estimated and highly significant coefficients. However, given that these were independently drawn samples, which were given similar but slightly different BWS designs, we had to allow for the potential for differences in scale between the two models. Therefore, rather than directly comparing the coefficients from the two models we looked at the ratio with respect to Control level 1 (which had a consistent definition in each instrument and was the most strongly estimated parameter in both models).

Figure 16 illustrates the extent of the consistency between the results obtained from the independent modelling of the two data sets, and shows that an R² of 0.992 is achieved when the scaled coefficient ratios from the two models are regressed against each other. Having established that the preferences were highly consistent (once the differences in model scale were taken into account) we decided to pool the data from the two survey waves. This necessitated the move to a scale heterogeneity MNL (S-MNL) model structure (using the naming convention developed by Fiebig et al. 93), which allows difference in model scale between data sets or individuals. In this case, the model was specified to allow heterogeneity in scale between the two data sets, allowing for the possibility that one of the data sets might contain less error than the other; however, in subsequent models this structure was also used to account for the difference in model scale between different groups of respondents.

FIGURE 16.

Comparison of relative parameter estimates from MOPSU and OSCA.

In order to take advantage of the pooled data, it was necessary to estimate separate coefficients for the different definitions used for accommodation cleanliness and comfort and food and drink in the MOPSU and OSCA data sets. The coefficients corresponding to the other domain levels could be jointly estimated from the two data sets. Most importantly, preliminary estimations showed that the model scale (amount of noise in the choice data) was not significantly different between the two data sets. As a result, this level of freedom was dropped from the model specification, which halved the number of utility functions and reduced the model run time.

Owing to an error in the MOPSU preference survey (see Table 1) development, and miscommunication between the market research agency and their interviewers, 190 people or 37% of the sample in the MOPSU survey were given a version of the BWS experiments where the order of the domains was also randomised within the BWS tasks for an individual. This is referred as ‘excessive randomisation’. There was more confusion (noise) among respondents who were given the excessive-randomisation BWS tasks, but the point estimates of domain-level weights were not, on the whole, significantly different, once the scale difference between the data sets was taken into account. 109 As the impact of the excessive randomisation was found to be relatively small, it was possible to specify the models with an additional scale parameter to allow for differences in the consistency, which these groups may have demonstrated in their choices within the exercises (with the a priori assumption from the MOPSU findings that those with greater randomisation in the task would have been less consistent in the reporting of their preferences).

Preliminary analysis with the pooled data set also found strong evidence of a difference in scale between ‘best’ and ‘worst’ choices, with people appearing to find it easier to identify the best, rather than the worst, aspect of the situations. As a result, the models reported below also control for potential differences in respondents’ ability to choose the ‘best’ and ‘worst’ domains. In addition to common problems, our prior expectation was that some respondents might find it easier to respond to BWS tasks than others. These differences between groups in the sample were likely to introduce more noise (higher variance) in the models, which needed to be captured using appropriate model specifications. This was confirmed through tests that indicated that there were differences associated with socioeconomic characteristics and place of residence. A detailed discussion of the rationale and final models’ specification is provided in Appendix 8.

Finally, the presentation of the BWS tasks in both the general population and the service user surveys involved randomisation of the order of domains between respondents in order to control for the impact that the position of the domain might have had on the likelihood of that domain being selected as the best or worst option. This randomisation ensured that, if the position of the domain was to induce any bias, all of the domains would suffer this bias equally and there would be no systematic over- or underestimation of the value of any given domain. By recording the position of each domain in any given BWS task, it was possible to incorporate this information within the parameterisation of the model and quantify the increase (or decrease) in the likelihood of the respondents choosing a given domain owing to its position independent of the estimation of the value placed on the level at which was presented. The final models contained eight position dummy-coded variables, which reflected the position of a domain level when that was chosen, contingent upon whether the respondent was asked to choose a ‘best’ or ‘worst’ option.

A generalised example of the utility function specification for the accommodation cleanliness and comfort domains is shown in Box 5.

BOX 5 - Generalised form of the utility function for the domain accommodation cleanliness and comfort

U_i (accommodation) =

+ β_accomOSCA × (1, if accommodation levela = 1)_i × (1, if choice scenario = best or second best)_i × (1, if OSCA data)

– β_accomOSCA × (1, if accommodation level = 1)_i × (1, if choice scenario = worst or second worst)_i × (1, if OSCA data)

+ β_accomOSCA × (1, if accommodation level = 2)_i × (1, if choice scenario = best or second best)_i × (1, if OSCA data)

– β_accomOSCA × (1, if accommodation level = 2)_i × (1, if choice scenario = worst or second worst)_i × (1, if OSCA data)

+ β_accomOSCA × (1, if accommodation level = 3)_i × (1, if choice scenario = best or second best)_i × (1, if OSCA data)

– β_accomOSCA × (1, if accommodation level = 3)_i × (1, if choice scenario = worst or second worst)_i × (1, if OSCA data)

+ β_accomOSCA × (1, if accommodation level = 4)_i × (1, if choice scenario = best or second best)_i × (1, if OSCA data)

– β_accomOSCA × (1, if accommodation level = 4)_i × (1, if choice scenario = worst or second worst)_i × (1, if OSCA data)

+ β_accomMOPSU × (1, if accommodation level = 1)_i × (1, if choice scenario = best or second best)_i × (1, if MOPSU data)

– β_accomMOPSU × (1, if accommodation level = 1)_i × (1, if choice scenario = worst or second worst)_i × (1, if MOPSU data)

+ β_accomMOPSU × (1, if accommodation level = 2)_i × (1, if choice scenario = best or second best)_i × (1, if MOPSU data)

– β_accomMOPSU × (1, if accommodation level = 2)_i × (1, if choice scenario = worst or second worst)_i × (1, if MOPSU data)

+ β_accomMOPSU × (1, if accommodation level = 3)_i × (1, if choice scenario = best or second best)_i × (1, if MOPSU data)

– β_accomMOPSU × (1, if accommodation level = 3)_i × (1, if choice scenario = worst or second worst)_i × (1, if MOPSU data)

+ β_accomMOPSU × (1, if accommodation level = 4)_i × (1, if choice scenario = best or second best)_i × (1, if MOPSU data)

– β_accomMOPSU × (1, if accommodation level = 4)_i × (1, if choice scenario = worst or second worst)_i × (1, if MOPSU data)

+ β_top × (1, if accommodation appeared in the first row)_i × (1, if choice scenario = best or second best)_i

+ β_{second top} × (1, if accommodation appeared in the second row)_i × (1, if choice scenario = best or second best)_i

+ β_{second bottom} × (1, if accommodation appeared in the eighth row)_i × (1, if choice scenario = best or second best)_i

+ β_bottom × (1, if accommodation appeared in the ninth row)_i × (1, if choice scenario = best or second best)_i

+ β_top × (1, if accommodation appeared in the first row)_i × (1, if choice scenario = best or second worst)_i

+ β_{second top} × (1, if accommodation appeared in the second row)_i × (1, if choice scenario = best or second worst)_i

+ β_{second bottom} × (1, if accommodation appeared in the eighth row)_i × (1, if choice scenario = best or second worst)_i

+ β_bottom × (1, if accommodation appeared in the ninth row)_i × (1, if choice scenario = best or second worst)_i

+ ε_i

where ε_i is the error term that captures the unobserved heterogeneity owing to differences among observations

The ‘0-otherwise’ statement was specified in all ‘if-statements’ of the utility functions, but has been omitted in the above boxes to save space in the document.

Best–worst scaling results

Overall preferences

General population

Table 18 presents the domain-level weights estimated for the general population using the combined OSCA and MOPSU data. The goodness-of-fit measure as indicated by the rho-squared value = 0.229 showed that the model performed relatively well. 111 (A rho-squared value between 0.25 and 0.30 of a discrete choice model is roughly equivalent to a value of between 0.75 and 0.80 of a linear regression model, which is considered a model with a good fit.) All domain levels were estimated relative to domain control at level 4, which was defined as the base level because it had the lowest utility: respondents felt that having no control over their daily life was the worst domain level of all those presented. All other domain levels are valued more highly so have positive weights compared with this. In the tables the model coefficients that relate to the wording used in the MOPSU instrument are shown in italic text.

TABLE 18 - Estimated parameters using general population OSCA and MOPSU data

For all scale parameters reported, the t-statistics reflect the hypothesis test of the scale being statistically different from 1.

The ranking of the domain-level weights within each domain was as expected, with weights at the top levels (i.e. levels representing better SCRQoL) higher than weights in the second, third and fourth levels within each domain.

Domain level	Coefficient	t-ratio (robust)
Accommodation cleanliness and comfort
1. My home is as clean and comfortable as I want	4.27	38.2
2. My home is adequately clean and comfortable	3.83	34.4
3. My home is not quite clean or comfortable enough	1.87	18.8
4. My home is not at all clean or comfortable	1.44	15.5
1. My home is as clean and comfortable as I want (MOPSU)	4.01	33.4
2. My home is adequately clean and comfortable (MOPSU)	3.77	33.0
3. My home is less than adequately clean or comfortable (MOPSU)	1.94	19.0
4. My home is not at all clean or comfortable (MOPSU)	1.47	16.5
Safety
1. I feel as safe as I want	4.37	37.6
2. Generally I feel adequately safe, but not as safe as I would like	2.23	24.8
3. I feel less than adequately safe	1.47	18.6
4. I don’t feel at all safe	0.558	7.9
Food and drink
1. I get all the food and drink I like when I want	4.35	37.7
2. I get adequate food and drink at OK times	3.82	32.3
3. I don’t always get adequate or timely food and drink	1.47	16.1
4. I don’t always get adequate or timely food and drink, and I think there is a risk to my health	0.932	10.2
1. I get all the food and drink I like when I want (MOPSU)	4.04	33.0
2. I get food and drink adequate for my needs (MOPSU)	3.85	33.1
3. I don’t get all the food and drink I need, but I don’t think there is a risk to my health (MOPSU)	1.76	18.8
4. I don’t get all the food and drink I need, and I think there is a risk to my health (MOPSU)	1.09	12.0
Personal care
1. I feel clean and am able to present myself the way I like	4.49	37.6
2. I feel adequately clean and presentable	3.88	35.6
3. I feel less than adequately clean or presentable	1.34	17.0
4. I don’t feel at all clean or presentable	0.982	13.4
Control over daily life
1. I have as much control over my daily life as I want	4.93	37.9
2. I have adequate control over my daily life	4.54	37.0
3. I have some control over my daily life, but not enough	2.69	25.9
4. I have no control over my daily life	0.00	Base level
Social participation and involvement
1. I have as much social contact as I want with people I like	4.31	36.8
2. I have adequate social contact with people	3.69	34.3
3. I have some social contact with people, but not enough	2.46	24.7
4. I have little social contact with people and feel socially isolated	1.22	16.2
Dignity
1. The way I’m helped and treated makes me think and feel better about myself	4.16	35.0
2. The way I’m helped and treated does not affect the way I think or feel about myself	3.13	29.6
3. The way I’m helped and treated sometimes undermines the way I think and feel about myself	1.48	18.2
4. The way I’m helped and treated completely undermines the way I think and feel about myself	1.33	17.1
Occupation and employment
1. I’m able to spend my time as I want, doing things I value or enjoy	4.72	37.5
2. I’m able to do enough of the things I value or enjoy with my time	4.56	36.4
3. I do some of the things I value or enjoy with my time, but not enough	2.8	28.3
4. I don’t do anything I value or enjoy with my time	0.843	12.5
Domain position in the BWS task
Top: respondent chose the domain that appeared first	0.151	4.5
Top two: respondent chose the domain that appeared second	0.0755	2.7
Bottom: respondent chose the attribute that appeared last	0.0121	0.4
Bottom two: respondent chose the attribute that appeared before the last	–0.034	–1.2
Scale parameters
µexcessive: MOPSU data with ‘excessive randomisation’	0.806	4.26a
µworst: data relate to a ‘worst’ or ‘second-worst’ choice	0.843	7.4
Model diagnostics
No. of observations	29,728
df	45
Final log-likelihood	–42,491.7
Rho2 (0)	0.229

The estimated weights in Figure 17 were all statistically significant, with relatively high t-ratios.

FIGURE 17.

Estimated weights and CIs in the general population model.

An examination of the coefficient estimates and their 95% CIs in Figure 17 revealed a small number of cases where the levels within the same domain were not statistically different from each other:

• food and drink between levels 1 and 2 in the MOPSU data set

• occupation between levels 1 and 2, and

• dignity between levels 3 and 4.

These findings imply that respondents either could not identify the differences in the wording, or felt that the outcomes of these domain levels were the same. However, it was encouraging that the food and drink weights at levels 1 and 2 in the OSCA experiment were significantly different, suggesting that the problem identified earlier had been successfully addressed by changes in the wording.

It was also informative to review the rank order of the estimated weights, as this ranking revealed the relative importance of each of the domain levels. As shown in Figure 17, the highest weight-value was estimated for the domain control over daily life at its top level (‘I have as much control over my daily life as I want’). Closely following control, respondents then preferred the top two levels of occupation and employment (‘I’m able to spend my time as I want’ and ‘I’m able to do enough of the things I value or enjoy with my time’). This ranking of domain levels meant that, if respondents were presented with a BWS task in which all domains were at their top levels, they were more likely to choose the top level of control over daily life. When comparing the middle levels of domains, respondents in the general population again placed the highest valuation on the domain control (‘I have adequate control over my daily life’), followed by personal care (‘I feel adequately clean and presentable’) and accommodation cleanliness and comfort. The least preferred levels were the lowest levels of safety, followed by occupation and employment, food and drink (OSCA) and personal care.

The parameters of the position variables capturing instances in which a domain was chosen when it appeared first or second from the top (i.e. top and top two) were also statistically significant. This finding implied that respondents were more likely to choose a domain (either as ‘best’ or ‘worst’) when it appeared first or second in the BWS task. The order implied by the coefficient values was intuitive; however, the overall effect of these variables as seen from the values of their coefficients was considerably smaller than the main effect of the domain weights, and the significance of these coefficients was also significantly lower. On the other hand, the parameters reflecting choices of domains when these appeared at the bottom or second from the bottom in a BWS task were not statistically significant.

Finally, the scale parameters µ_worst and µ_excessive were both significantly different from 1. The significance of the scale parameter µ_worst implied that the variance (noise) between ‘best’ (including ‘best’ and ‘second best’) and ‘worst’ (including ‘worst’ and ‘second worst’) responses was significantly different. Having set the scale of ‘best’ choices equal to 1 for identification, the estimated value of scale equal to 0.843 implied higher variance in the ‘worst’ choices, which meant that the data from the ‘worst’ responses generated a less-deterministic choice process than the ‘best’ responses. Practically, this meant that respondents placed more emphasis on the domain levels when they indicated their ‘best’ rather than their ‘worst’ choices. The significance of the scale parameter µ_excessive suggested that the data from those respondents facing excessive randomisation of BWS tasks had a less-deterministic choice process than those facing the intended randomisation of the domain order, which was held constant for a given respondent. Again, the value of the scale parameter being < 1 implied that the ‘excessive’ randomisation data in the MOPSU data set contained more noise than the data with ‘intended’ randomisation.

Service users

The domain-level weights estimated from the service users’ BWS data are presented in Table 19. As in the general population models, the domain control at level 4 was set as the base level. All domain weights were statistically significant, and the model diagnostics showed that the service users’ model also performed well.

TABLE 19 - Estimated parameters using service user data

Domain level	Coefficient	t-ratio (robust)
Accommodation cleanliness and comfort
1. My home is as clean and comfortable as I want	4.50	22.1
2. My home is adequately clean and comfortable	4.00	20.8
3. My home is not quite clean or comfortable enough	2.20	13.2
4. My home is not at all clean or comfortable	1.80	11.3
Safety
1. I feel as safe as I want	4.66	22.5
2. Generally I feel adequately safe, but not as safe as I would like	2.57	15
3. I feel less than adequately safe	1.62	11.1
4. I don’t feel at all safe	0.72	5.4
Food and drink
1. I get all the food and drink I like when I want	4.34	21.4
2. I get adequate food and drink at OK times	4.04	21.4
3. I don’t always get adequate or timely food and drink	1.96	12.9
4. I don’t always get adequate or timely food and drink, and I think there is a risk to my health	1.19	9.3
Personal care
1. I feel clean and am able to present myself the way I like	4.88	23.0
2. I feel adequately clean and presentable	4.09	20.3
3. I feel less than adequately clean or presentable	1.55	10.2
4. I don’t feel at all clean or presentable	1.16	8.2
Control over daily life
1. I have as much control over my daily life as I want	5.29	23.2
2. I have adequate control over my daily life	4.71	21.4
3. I have some control over my daily life, but not enough	3.09	15.2
4. I have no control over my daily life	0.00	Base level
Social participation and involvement
1. I have as much social contact as I want with people I like	4.59	21.9
2. I have adequate social contact with people	3.90	19.1
3. I have some social contact with people, but not enough	2.92	14.4
4. I have little social contact with people and feel socially isolated	1.49	10.1
Dignity
1. The way I’m helped and treated makes me think and feel better about myself	4.45	21.2
2. The way I’m helped and treated does not affect the way I think or feel about myself	3.70	18.5
3. The way I’m helped and treated sometimes undermines the way I think and feel about myself	1.88	12.5
4. The way I’m helped and treated completely undermines the way I think and feel about myself	1.63	10.7
Occupation and employment
1. I’m able to spend my time as I want, doing things I value or enjoy	5.12	24.2
2. I’m able do enough of the things I value or enjoy with my time	4.74	21.9
3. I do some of the things I value or enjoy with my time, but not enough	3.32	16.4
4. I don’t do anything I value or enjoy with my time	1.17	8.2
Domain position in the BWS task
Top: respondent chose the domain that appeared first	0.272	5.2
Top two: respondent chose the domain that appeared second	0.138	3.1
Bottom: respondent chose the attribute that appeared last	0.0385	0.8
Bottom two: respondent chose the attribute that appeared before the last	–0.0451	–1.0
Scale parameter
µworst: data relate to a ‘worst’ or ‘second-worst’ choice	0.6862	10.0
Model diagnostics
No. of observations	11,744
df	36
Final log-likelihood	–17,042.5
Rho2 (0)	0.217

Figure 18 shows the ranking of domain-level weights within domains, which appeared to be consistent and agreed with prior expectations, with weights at top levels higher than weights in the second, third and fourth levels. Within domains, there were three cases where the levels within the same domain were not statistically different at the 95% level:

• food and drink between levels 1 and 2

• occupation and employment between levels 1 and 2

• dignity between levels 3 and 4.

FIGURE 18.

Estimated weights and CIs in the service users’ model.

The rank order of the estimated weights, as seen from Figure 18, revealed that control over daily life at level 1 (‘I have as much control over my daily life’) was also the aspect of SCRQoL that was valued the highest by service users. The second and third highest valued domains were occupation and employment and personal care, both at level 1. Occupation and employment and control over daily life at level 2 were the highest-valued middle-level domains and, overall, among the top five valued domains. Service users valued least the lowest levels of safety (‘I don’t feel at all safe’), personal care (‘I don’t feel at all clean or presentable’), occupation and employment (‘I don’t do anything I value or enjoy with my time’) and food and drink (‘I don’t always get adequate or timely food and drink’, and ‘I think there is a risk to my health’).

The parameters of the position variables were significant only in the case that the chosen domain appeared first or second in the BWS task. Service users were also more likely to choose a domain that appeared first or second in the domain list. The order implied by the value of the coefficients of the top and top two parameters was intuitive; however, the main effect of the domain-level weights and the significance of these coefficients were significantly lower. These findings were in line with the findings in the general population models.

At this stage, the service users’ model included a single scale to account for potential differences between ‘best’ and ‘worst’ choices, as no service user had faced excessive randomisation. The estimated scale parameter µ_worst was statistically significantly different with respect to 1 (t-ratio = 10, see Table 20). This finding implies that there was more noise (higher variance) in the ‘worst’ choices relative to ‘best’ choices.

TABLE 20 - Estimated parameters using general population OSCA and MOPSU data with scale effects

Domain level	Coefficient	t-ratio (robust)
Accommodation cleanliness and comfort
1. My home is as clean and comfortable as I want	5.06	27.3
2. My home is adequately clean and comfortable	4.57	25.9
3. My home is not quite clean or comfortable enough	2.19	16.8
4. My home is not at all clean or comfortable	1.68	14.1
1. My home is as clean and comfortable as I want (MOPSU)	4.76	25.9
2. My home is adequately clean and comfortable (MOPSU)	4.47	25.6
3. My home is less than adequately clean or comfortable (MOPSU)	2.28	17.1
4. My home is not at all clean or comfortable (MOPSU)	1.73	15.0
Safety
1. I feel as safe as I want	5.16	27.1
2. Generally I feel adequately safe, but not as safe as I would like	2.65	20.3
3. I feel less than adequately safe	1.75	16.3
4. I don’t feel at all safe	0.67	7.7
Food and drink
1. I get all the food and drink I like when I want	5.15	27.4
2. I get adequate food and drink at OK times	4.54	25.1
3. I don’t always get adequate or timely food and drink	1.72	14.5
4. I don’t always get adequate or timely food and drink, and I think there is a risk to my health	1.08	9.8
1. I get all the food and drink I like when I want (MOPSU)	4.77	25.2
2. I get food and drink adequate for my needs (MOPSU)	4.55	24.9
3. I don’t get all the food and drink I need, but I don’t think there is a risk to my health (MOPSU)	2.09	16.6
4. I don’t get all the food and drink I need, and I think there is a risk to my health (MOPSU)	1.26	11.4
Personal care
1. I feel clean and am able to present myself the way I like	5.34	26.9
2. I feel adequately clean and presentable	4.62	25.8
3. I feel less than adequately clean or presentable	1.55	15.3
4. I don’t feel at all clean or presentable	1.14	12.6
Control over daily life
1. I have as much control over my daily life as I want	5.86	27.1
2. I have adequate control over my daily life	5.39	26.7
3. I have some control over my daily life, but not enough	3.17	21.3
4. I have no control over my daily life	0.00	Base level
Social participation and involvement
1. I have as much social contact as I want with people I like	5.12	27.2
2. I have adequate social contact with people	4.38	25.8
3. I have some social contact with people, but not enough	2.91	20.1
4. I have little social contact with people and feel socially isolated	1.41	14.6
Dignity
1. The way I’m helped and treated makes me think and feel better about myself	4.96	25.6
2. The way I’m helped and treated does not affect the way I think or feel about myself	3.73	22.7
3. The way I’m helped and treated sometimes undermines the way I think and feel about myself	1.73	16.6
4. The way I’m helped and treated completely undermines the way I think and feel about myself	1.54	15.7
Occupation and employment
1. I’m able to spend my time as I want, doing things I value or enjoy	5.64	26.5
2. I’m able do enough of the things I value or enjoy with my time	5.43	26.1
3. I do some of the things I value or enjoy with my time, but not enough	3.32	22.3
4. I don’t do anything I value or enjoy with my time	1.00	11.9
Domain position in the BWS task
Top: respondent chose the domain that appeared first	0.18	4.6
Top two: respondent chose the domain that appeared second	0.08	2.5
Bottom: respondent chose the attribute that appeared last	0.02	0.4
Bottom two: respondent chose the attribute that appeared before the last	–0.04	–1.2
Scale parameters
µexcessive: MOPSU data with ‘excessive randomisation’	0.82	3.7
µworst: data relate to a ‘worst’ or ‘second-worst’ choice	0.85	7.0
µnot-imagine: data relate to individuals who could not put themselves in the imaginary positions described in the choices	0.79	3.2
µLondon and South-East: data relate to individuals living in London and the South East	0.82	5.3
µbhealth: data relate to individuals with self-reported fair, bad or very bad health	0.87	3.3
µint. duration < 15 min:data relate to individuals’ interview duration shorter than 15 minutes	0.87	2.6
Model diagnostics
No. of observations	29,728
df	49
Final log-likelihood	–42,327.9
Rho2 (0)	0.233

Other factors associated with variance heterogeneity

A second set of models was estimated in which we further explored the potential influence of other scaling effects, known as unobserved or variance heterogeneity, in both the general population and service user data. Variances may differ for a number of reasons. The most intuitive reason for the presence of variance heterogeneity is that different people may have different degrees of certainty in their answers in the BWS task. Failure to model differences in variances leads to bias in the point estimates (utility weights),93,94 and Flynn et al. 96 discuss this with reference to QoL data. The risk is that variations in preferences can be attributed to groups, which are about differences in scale consistency (or certainty) rather than differences in actual preferences.

In undertaking the analysis, we made a number of hypotheses of what we might expect to affect variations in scale consistency. In particular, we anticipated that scale consistency could be influenced by or associated with:

• education

• age

• experience of services and providing care

• current level of QoL

• location

• interviewer

• length of time taken during the exercises

• a person’s ability to relate to the situation.

General population

A series of model specifications was tested prior to arriving at the final model specification, which is reported in Table 20. The initial set of analyses investigated each of the potential associations of scale consistency in the model. The statistically significant scale parameters were added to the final model. The specification of the model with respect to domain levels and position-related variables remained the same as in the model reported in Table 18.

The scale parameters reported in the model shown in Table 20 suggested that, in addition to the previous groups identified, four further subgroups of respondents in the general population sample made less deterministic choices that their counterparts:

• respondents who could not put themselves in the imaginary situations described

• respondents who lived in London or the south-east of England

• respondents with fair, bad or very bad self-reported health status

• where the interview time was shorter than 15 minutes.

Figure 19 shows that the ranking of domain levels within the domains was retained. As expected, the values of the domain-level weights increased relative to the model with fewer scale parameters. This was expected because the remaining unobserved variance in the model became smaller after introducing the additional scale parameters.

FIGURE 19.

Estimated weights and CIs in the general population model.

In this model, however, we did notice an additional overlap between levels 1 and 2 of the domain accommodation cleanliness and comfort, which were previously found to be marginally different from each other. The overlaps between food and drink (MOPSU), occupation and employment at levels 1 and 2, and dignity at levels 3 and 4 were retained.

The overall rank order of the estimated domain weights between the scale effects and limited scale effects models was retained, and the significance of the position parameters when the chosen domain appeared first or second in the BWS task was retained. Again, the estimates were intuitive and implied that a domain was more likely to be chosen if it appeared first or second. However, the main effect and significance of these variables were less than that of the domain levels.

Service users

The specification tests for potential scale effects in the service users’ sample revealed two further subgroups of respondents who made fewer deterministic choices:

• residents outside the south-east of England, and

• respondents who had ‘no formal qualification’ or just ‘GCSE/O-level’.

Figure 20 shows that the ranking of domain levels within the domains was retained. As in the general population scale-effect model, the value of each domain-level weight increased compared with the limited scale effects model.

FIGURE 20.

Estimated weights and CIs in the general population limited scale effects model.

In the scale effects model, the top two and bottom two levels of the domain accommodation cleanliness and comfort were similarly valued by the respondents and were not statistically different, respectively. As with the limited scale effects model, the top two levels of food and drink and occupation and employment and the bottom two levels of dignity were not significantly different in this scale effects model.

With a few minor variations, the overall ranking of the domain-level weights found in the limited scale effects model was broadly retained, and the position parameters when the chosen domain appeared first or second in the BWS task were statistically significant as in the limited scale effects models.

Comparison of domain weights between the general population and service users

An important question to address was whether service users’ preferences differed from those of the general population and, if so, whether or not these differences suggested alternative utility weights. We compared the results of the scale effects models, reported in Tables 20 and 21. These models had different scales, so the coefficients were not directly comparable. 105 However, we could look to the relative size of the differences using one of the domain levels as a common denominator and normalising all others relative to that. We chose the top level of the domain control, which was strongly estimated in both the general population and the service user models.

TABLE 21 - Estimated parameters using service users’ data with scale effects

Domain level	Coefficient	t-ratio (robust)
Accommodation cleanliness and comfort
1. My home is as clean and comfortable as I want	6.48	13.6
2. My home is adequately clean and comfortable	5.74	13.1
3. My home is not quite clean or comfortable enough	3.14	9.9
4. My home is not at all clean or comfortable	2.58	9.3
Safety
1. I feel as safe as I want	6.79	13.0
2. Generally I feel adequately safe, but not as safe as I would like	3.60	10.9
3. I feel less than adequately safe	2.25	8.9
4. I don’t feel at all safe	1.01	5.2
Food and drink
1. I get all the food and drink I like when I want	6.25	13.0
2. I get adequate food and drink at OK times	5.81	13.0
3. I don’t always get adequate or timely food and drink	2.78	10.4
4. I don’t always get adequate or timely food and drink, and I think there is a risk to my health	1.59	7.8
Personal care
1. I feel clean and am able to present myself the way I like	7.02	13.7
2. I feel adequately clean and presentable	5.87	12.8
3. I feel less than adequately clean or presentable	2.30	8.8
4. I don’t feel at all clean or presentable	1.72	7.3
Control over daily life
1. I have as much control over my daily life as I want	7.64	13.3
2. I have adequate control over my daily life	6.74	12.8
3. I have some control over my daily life, but not enough	4.42	11.2
4. I have no control over my daily life	0.00	Base level
Social participation and involvement
1. I have as much social contact as I want with people I like	6.66	13.2
2. I have adequate social contact with people	5.59	12.2
3. I have some social contact with people, but not enough	4.11	10.8
4. I have little social contact with people and feel socially isolated	2.05	8.4
Dignity
1. The way I’m helped and treated makes me think and feel better about myself	6.48	12.9
2. The way I’m helped and treated does not affect the way I think or feel about myself	5.41	12.0
3. The way I’m helped and treated sometimes undermines the way I think and feel about myself	2.68	10.2
4. The way I’m helped and treated completely undermines the way I think and feel about myself	2.30	9.4
Occupation and employment
1. I’m able to spend my time as I want, doing things I value or enjoy	7.39	13.5
2. I’m able do enough of the things I value or enjoy with my time	6.84	13.0
3. I do some of the things I value or enjoy with my time, but not enough	4.75	11.4
4. I don’t do anything I value or enjoy with my time	1.60	7.4
Domain position in the BWS task
Top: respondent chose the domain that appeared first	0.37	4.9
Top two: respondent chose the domain that appeared second	0.17	2.6
Bottom: respondent chose the attribute that appeared last	0.04	0.6
Bottom two: respondent chose the attribute that appeared before the last	–0.06	–0.8
Scale parameters
µworst: data relate to a ‘worst’ or ‘second-worst’ choice	0.69	9.5
µnotSE: data relate to residents outside the south-east of England	0.66	7.9
µO-level: data relate to individuals with no formal qualification or GCSE/O-level	0.83	2.9
Model diagnostics
No. of observations	11,744
df	38
Final log-likelihood	–16,914.7
Rho2 (0)	0.224

Figure 21 shows the relative values and the 95% CIs (see Hess and Daly106 for computation of SEs).

FIGURE 21.

Rescaled domain-level weights between the general population and service users.

We used t-tests to examine whether or not the normalised preference weights for each domain level differed significantly between the general population and service users. Table 22 shows that preference weights were very similar among the general population and service users and in most cases were not statistically different. Only in 6 out of 30 domain-level weights were the valuations between the general population and service users significantly different at the 95% level.

TABLE 22 - Comparison of estimated parameters between OSCA general population and service users

Domain level	General population		Service user		Significant at 95% CI
	Coefficient	SE	Coefficient	SE
Accommodation cleanliness and comfort
1. My home is as clean and comfortable as I want	0.86	0.014	0.85	0.017	No
2. My home is adequately clean and comfortable	0.78	0.013	0.75	0.018	No
3. My home is not quite clean or comfortable enough	0.37	0.016	0.41	0.021	No
4. My home is not at all clean or comfortable	0.29	0.016	0.34	0.021	No
Safety
1. I feel as safe as I want	0.88	0.012	0.89	0.016	No
2. Generally I feel adequately safe, but not as safe as I would like	0.45	0.012	0.47	0.022	No
3. I feel less than adequately safe	0.30	0.012	0.29	0.022	No
4. I don’t feel at all safe	0.11	0.013	0.13	0.022	No
Food and drink
1. I get all the food and drink I like when I want	0.88	0.014	0.82	0.015	Yes
2. I get adequate food and drink at OK times	0.77	0.014	0.76	0.015	No
3. I don’t always get adequate or timely food and drink	0.29	0.015	0.36	0.019	Yes
4. I don’t always get adequate or timely food and drink, and I think there is a risk to my health	0.18	0.017	0.21	0.021	No
Personal care
1. I feel clean and am able to present myself the way I like	0.91	0.011	0.92	0.016	No
2. I feel adequately clean and presentable	0.79	0.011	0.77	0.017	No
3. I feel less than adequately clean or presentable	0.27	0.013	0.30	0.021	No
4. I don’t feel at all clean or presentable	0.19	0.013	0.23	0.022	No
Control over daily life
1. I have as much control over my daily life as I want	1.00 (normalised level)
2. I have adequate control over my daily life	0.92	0.010	0.88	0.016	Yes
3. I have some control over my daily life, but not enough	0.54	0.014	0.58	0.025	No
4. I have no control over my daily life	0.00 (base level)
Social participation and involvement
1. I have as much social contact as I want with people I like	0.87	0.010	0.87	0.014	No
2. I have adequate social contact with people	0.75	0.011	0.73	0.018	No
3. I have some social contact with people, but not enough	0.50	0.013	0.54	0.024	No
4. I have little social contact with people and feel socially isolated	0.24	0.012	0.27	0.022	No
Dignity
1. The way I’m helped and treated makes me think and feel better about myself	0.85	0.011	0.85	0.016	No
2. The way I’m helped and treated does not affect the way I think or feel about myself	0.64	0.012	0.71	0.016	Yes
3. The way I’m helped and treated sometimes undermines the way I think and feel about myself	0.30	0.013	0.35	0.020	Yes
4. The way I’m helped and treated completely undermines the way I think and feel about myself	0.26	0.013	0.30	0.021	No
Occupation and employment
1. I’m able to spend my time as I want, doing things I value or enjoy	0.96	0.011	0.97	0.014	No
2. I’m able do enough of the things I value or enjoy with my time	0.93	0.011	0.90	0.014	No
3. I do some of the things I value or enjoy with my time, but not enough	0.57	0.012	0.62	0.022	Yes
4. I don’t do anything I value or enjoy with my time	0.17	0.012	0.21	0.022	No

Respondents in the general population sample placed higher value on attaining the top levels of food and drink (‘I get all the food and drink I like when I want’) and control (‘I have as much control over my daily life as I want’) than service users. On the other hand, service users placed higher value on the top level of dignity (‘The way I’m helped and treated makes me think and feel better about myself’), compared with the general population. At the lower levels, it appeared that service users were slightly less concerned about being in a situation where:

• ‘I don’t always get adequate or timely food and drink’ (food and drink at level 3),

• ‘The way I’m helped and treated sometimes undermines the way I think and feel about myself ‘ (dignity at level 3), or

• ‘I do some of the things I value or enjoy with my time, but not enough’ (occupation and employment at level 3).

However, none of the observed differences in value was large in absolute terms and in testing so many differences we would expect some differences to appear statistically significant by chance. Our conclusion was that there was insufficient evidence to warrant any separate estimates of utility weights to reflect the preferences of service users.

Time trade-off study

We identified in Chapter 4 that the aim of the TTO exercise was to enable us to adjust the domain-level weights (from the BWS data) to obtain QALY-equivalent weights for social care on a scale where ‘0’ represents ‘being dead’ and ‘1’ represents the social care equivalent of ‘perfect health’.

Analysis

The TTO score for each individual for each state was calculated using a scale of ‘0’ (‘being dead’) to ‘1’ (‘all needs met’). For the cases where respondents valued states between these points, the usual linear TTO scoring scale was used to assign the score, i.e. the formula x/10 where x is the number of years spent with ‘all needs met’. For states that were rated as worse than being dead, the convention developed by Dolan112 was adopted, where the score is given by the formula (x/10) − 1, bounding the values to –1.

From the TTO data, the mean TTO score for each of the 64 states was calculated to provide the average value assigned to each state. These average scores were used on the basis that we were seeking to understand the relationship between the TTO score and the corresponding BWS weight, with the latter calculated from models providing the average weights utility estimated across the sample. The average TTO score for a given state was then plotted against the corresponding weight for that state as implied by the BWS modelling, by summing across the domain levels that described the state in question (using BWS domain levels weights on a scale of 0–1 as reported in Table 25). This plot showed that the relationship between the mean TTO score and the BWS weight was linear, and some of the SCRQoL states were rated as worse than being dead (TTO score < 0).

A simple mapping function form was estimated using

^{[Equation 1]} OLS : T T O ij = f ( D C E ) + ε ij

The first specification assumed a linear relationship with an intercept, and then squared and cubic terms were added to see whether or not performance is improved.

The regression was constrained to ensure that the ‘all needs met’ state had a value of 1.0 on the TTO scale, which is implied by how the TTO exercise is scored. The equation from this linear regression provides the transformation required to anchor the BWS weight to being dead, and provides a score that could be used to estimate a ‘SC-QALY’, which we have termed a SC-QALY index. The relationship between the two scoring schemes is shown in Figure 22. The results suggested that a simple linear relationship was sufficient, and indeed adding squared and cubic terms did not improve the relationship. There was a good overall fit of the model with an adjusted R² of 0.859, a root mean squared error of 0.124 and mean absolute error across SCRQoL states of 0.091. It is worthy of note that this is the first occasion that we are aware of a TTO exercise being successfully applied to social care, and the extent of the linearity in the sample level preferences obtained from the TTO and BWS tasks suggests that the TTO may be usefully exploited to further understand the temporal dimension of the population preferences.

FIGURE 22.

Regression of mean TTO score against BWS weight for 64 SCRQoL states. SC-QALY = (0.203 × BWS_weight) – 0.466.

The regression between the TTO and BWS scores implies that the pit state, where all domains are at the high needs level, is valued at –0.171 on the TTO scale. To give some context to this, the value that Dolan112 estimated for the health EQ-5D pit state (using a comparable scoring scheme for states worse than being dead) was –0.543.

Conclusion

The preference study work involved a number of different data collections from both the general population and service users. The results have shown a consistent picture in terms of both what we would hope to observe (higher values being put on higher SCRQoL states according to our definitions) and patterns of actual preferences. This has given us confidence in generating utility weights based on these analyses for our measure. In the next chapter, we turn to the final measure itself.

Introduction

The aim of the study was to develop a preference-weighted measure of social care outcomes that could be used for a wide variety of purposes. We built on previous work and took opportunities beyond this project to ensure that the measure was as well founded as possible. The previous chapters have described the methodological challenges and how these were met, both in developing the instrument and in identifying an anchored set of preference weights, and in the process investigating factors associated with these. In this final chapter, we provide an overview of the final measure, describing the distributions and demonstrating the application of the utility weights. We also call attention to some limitations of the study, discuss the potential uses and value of the measure, and identify future work needed.

Population and service user item distributions

As part of the interviews undertaken to establish preferences of the general population and service users, respondents were asked about their current SCRQoL state in order both to familiarise them with the domains and levels, and to inform the analysis. This also provides us with useful comparative data on levels of SCRQoL in these two groups.

Table 26 shows the responses to the individual items. In making comparisons, it should be borne in mind that service users were all people who had received equipment services, so their needs would predominantly be related to physical impairments rather than mental health problems. Moreover, they all were sufficiently cognitively able to undertake the BWS interview. As a result, their reported SCRQoL states are not necessarily representative of service users as a whole. The dignity question was asked of members of the general population only if they were currently receiving any help, which could be in the form of services or informal care. It was interesting that this applied to nearly one-quarter (24%) of the general population sample.

With the exception of dignity, members of the general population were significantly more likely to report the ‘ideal’ state and, with the exception of occupation, service users were significantly more likely to report the ‘no needs’ (i.e. where needs are met but not to the desired level), as well as ‘needs’ states, in each domain. This is to be expected if services are just meeting people’s needs rather than further promoting capability, and suggests that the measure is sensitive to the distinction we were aiming to achieve by the additional level (see Chapter 2). The distribution of the occupation domain may reflect the nature of the general population sample which, although generally reasonably representative of the national population, had higher proportions of people who were retired (see Appendix 6). Unlike the other domains, the distribution of responses to dignity from members of the population who were receiving help was very similar to that of service users, although they were more likely to report that the way the help was provided made them feel better about themselves.

Utility weighting

Chapter 5 described the preference study analyses, providing a range of insights into the preferences of both the general population and service users with respect to SCRQoL. The principal objective, however, was to develop a utility weighting system that could be used for assessing different care interventions. It is useful, therefore, to review the steps required to apply the findings from the study to obtain the ‘SC-QALY’.

The first step is to assess the SCRQoL state using the domain levels shown in Table 27.

For example, the individual may be assessed to have the following profile across the domains:

• accommodation cleanliness and comfort – level 2 ‘My home is adequately clean and comfortable’

• safety – level 3 ‘I feel less than adequately safe’

• food and drink – level 1 ‘I get all the food and drink I like when I want’

• personal care – level 2 ‘I feel adequately clean and presentable’

• control over daily life – level 3 ‘I have some control over my daily life, but not enough’

• social participation and involvement – level 4 ‘I have little social contact with people and feel socially isolated’

• dignity – level 3 ‘The way I’m helped and treated sometimes undermines the way I think and feel about myself’

• employment and occupation – level 2 ‘I’m able do enough of the things I value or enjoy with my time.’

This defines their SCRQoL state.

The state can then be scored by using the weights provided in Table 28, which, as Chapter 5 showed, reflect the preferences of both the general population and service users.

In the case of this example, the domain level weights are:

• accommodation cleanliness and comfort – level 2 domain level score = 0.780

• safety – level 3 domain level score = 0.298

• food and drink – level 1 domain level score = 0.879

• personal care – level 2 domain level score = 0.789

• control over daily life – level 3 domain level score = 0.541

• social participation and involvement – level 4 domain level score = 0.241

• dignity – level 3 domain level score = 0.295

• employment and occupation – level 2 domain level score = 0.927.

This leads to the following calculation for the state score:

This state score allows a comparison of the relative value placed on this state compared with other states, and for many applications will be sufficient. However, if desired, the state score can be translated into a score that could be used to estimate a ‘SC-QALY’ score, using the equation developed from the regression of TTO score against the BWS weight in Chapter 5:

which in this case gives a state with the following SC-QALY score:

This provides an equivalent to the widely used measures of HRQoL, such as EQ-5D used as a basis for health QALYs. Although more work will be required to determine the extent to which this score can be used alongside or compared with the long-established QALY, the score does provide the analyst with the value associated with the given SCRQoL state on a scale where ‘0’ = ‘being dead’ and ‘1’ = ‘best’ or ‘ideal’ SCRQoL.

Where the interest is in establishing the ‘expected’ or ‘gain’ in SCRQoL, the same weights are applicable, as the questions used to establish ‘expected’ SCRQoL states use the same descriptors. Thus, in the example above, if an individual had high (level 4) needs for personal cleanliness and food and drink in the absence of services, his/her expected SCRQoL would be measured as:

which can be deducted from the current score to provide a measure of SCRQoL gain:

Population and service user distributions of the measure

For each SCRQoL state, three different summary scores can be produced:

• One score assumes equal weights for domains and response options and takes values from 0 to 24.

• Each level and domain can be weighted using the valuations from the BWS preference elicitation study to produce a current SCRQoL index, which can take values from 1.46 to 7.22.

• The SC-QALY index takes the BWS values and anchors these to the ‘being dead’ state at 0 as described above. This can take values from –0.17 to 1 (or 0.999 when rounded to three decimal places).

The distributions of these scores for both the equally weighted and preference-weighted current SCRQoL indices for the general population and service user population are shown in Table 29. Figures 23 and 24 show the distributions of the SC-QALY index for these populations graphically. As might be hoped, no members of the population or service user group reported current SCRQoL states that would be rated as poorer than ‘being dead’. Despite the compensatory activity of services, our measure suggests that SCRQoL is significantly better for the general population than for users of equipment services [using the SC-QALY index, F(1, 956) = 169.71, p < 0.001].

FIGURE 23.

Distribution of the SC-QALY index for service user population sample (n = 458). Kernel = Epanechnikov; bandwidth = 0.0492.

FIGURE 24.

Distribution of the SC-QALY index for general population sample (n = 500). Kernel = Epanechnikov; bandwidth = 0.0256.

Validity and reliability

In Chapter 3 we explored the construct validity of the descriptive system of the provisional measure with a sample of older home care service users. Although the preference study survey with users of equipment services was not designed specifically for exploring validity, the data collected did allow us to investigate many of the associations between sociodemographic, health and functional ability-related characteristics with a service user group that reflected a wider age range. Analysis of individual items that had been amended yielded a consistent pattern of results (see Appendix 4). A single item reporting QoL was significantly associated with the combined SCRQoL equally weighted and preference-weighted scores (R² = 0.57; p < 0.001).

We investigated reliability through a follow-up survey of a subset of the equipment service users who participated in the BWS interview. Unfortunately, problems in conducting the fieldwork meant that the intended follow-up period of about 21 days was in practice over 74 days (SD = 18.6 days, max = 116 days, min = 25 days, n = 101). We ran a variance components model to estimate the intraclass correlation coefficient (ICC) for the matched sample. The dependent variable was SCRQoL and the cluster variable the person. The reliability estimate was 0.68 when, ideally, we would want a reliability of > 0.8. The analysis was repeated splitting the sample between those who had the follow-up interview more or less than 60 days after the first interview. The ICC for the former was 0.65 and for the latter 0.75, implying that, had the guidance been followed and a shorter time period used, better test–retest reliability might have been observed. Under restricted maximum likelihood (REML) estimation for those interviewed within 60 days, n = 50, groups = 25; for those over 60 days, n = 150, groups = 75. The small number of groups for the shorter time period means that the estimates for the ICC may not be that accurate, but the results are indicative.

The indications were generally very positive with respect to the validity and reliability of the estimated preference weights. Chapter 5 describes models that yielded consistent results, with higher utilities associated with higher levels of SCRQoL in each domain. Associations with preferences were logically consistent where we might have had prior expectations, and the results were remarkably stable. Very similar models were obtained from different general population samples taken a year apart and from service users in different geographical areas. The feasibility study demonstrated very similar results using DCE and BWS. 105 The preference weights on our final model using the TTO method were highly correlated with and demonstrated a strong linear relationship with BWS estimates (see Chapter 5). Those factors found to be associated with variations in preferences were not ones (such as employment status) where our population sample differed from the general population. Thus, it is with some confidence that we can use the models to estimate our preference weights.

We have focused primarily on our measure of current SCRQoL state, but also noted that the compensatory nature of social care makes it particularly difficult to identify outcomes in practice. We have proposed a pragmatic approach that builds on this characteristic to allow individuals to identify the alternative, using the difference to generate a measure of SCRQoL ‘gain’. The method is similar to the retrospective pre-test design proposed as a method for minimising the effects of response shift in other areas where such problems are encountered. 96,113–116 There is some evidence for the validity of the approach from this study (see Chapter 3), and with previous versions for day care7,9 and care homes,8 but more work is needed if it is to be widely applied.

Limitations

Necessarily, there are a number of limitations to the study. In Chapter 1 we described the nature of social care and identified the key role of the informal care network, with informal carers acting as both providers and beneficiaries of care. If we want to comprehensively reflect the impact of social care, we should be measuring outcomes for carers as well as for individuals with impairments. Our measure, although covering aspects of QoL that are relevant to everyone, reflects the objectives of services in relation to people with impairments.

As described in Chapter 2, our measure built on previous work on outcomes for older people and then developed to cover the range of domains relevant to younger age groups. Each domain covers a range of different aspects and it is feasible that there are other ways of describing the domains which might be more suited to specific groups. For example, a measure focused on younger adults with mental health problems might bring together more basic aspects of outcome, which are less frequently relevant to this group, and further divide domains relating to social participation and occupation.

However, the cognitive interviews did establish the applicability and relevance of all the domains to the full range of service user groups, confirmed by item response rates from different user groups,11 and we were able to test the measure on users of equipment services, who include the full range of ages. Moreover, there were no indications that any domains were seen as irrelevant or any age-related variations in preferences. Although different SCRQoL measures might be better suited for certain groups, older people constitute the majority of social care service users, so it is appropriate that a generic measure ensures that the main areas of social care intervention relevant to this group are fully reflected.

When testing the measure, our service user samples were dependent on the UESs being conducted at the time of the study. This meant that much of the validity work was confined to older home care service users, and our service user survey samples were not ethnically diverse, although our population samples did have a good proportion of people from BME groups. Although we drew on data where they were available (e.g. from the IBSEN study49), for the most part it was not possible to involve people with cognitive impairments because of the cognitively demanding nature of much of the work.

We have investigated the descriptive validity of our items and the scales, and presented some evidence for the validity of the preference weights, but Brazier et al. 70 also identify empirical validity as a desirable characteristic of a preference-weighted measure. This has not been possible within the scope of this study. Furthermore, although there is no evidence of lack of reliability, our attempts to provide robust evidence of reliability did not meet with success.

As argued in Chapter 1, we might expect adaptation to be a particularly relevant issue in social care. The way the SCRQoL measure is constructed would lead us to expect that it measures to some extent, and in so far as adaptation occurs, the adapted utility of service recipients. The effect of adaptation may be (partially) mitigated in two ways. First, in using general population preferences, the utility weights associated with different states will be much less affected by adaptation; general population respondents are considering hypothetical situations rather than actually experiencing limited capacity, and we expect their ability to predict their own adaptation to be limited. Second, the descriptive content of the instrument emphasises an aspiration outcome level rather than just ‘usual’ or normal levels of activity, and this may allow respondents to think beyond their own immediate situation or circumstances.

Nonetheless, we do anticipate that SCRQoL would be affected if people do adapt, and that effect is likely to lead to measuring a lower impact of services than if adaptation did not occur. It would be a useful development in the future to attempt to get some empirical estimate of the size of adaptation effects when using the instrument. While accepting that adaptation effects may occur, it is not at all clear whether, normatively, we should be measuring adapted utility or making allowances for adaptation where that is possible. The theoretical debate on this subject appears divided. 44 On the one hand, not adjusting for adaptation will lead to a lower priority being given to services and service recipients where adaptation occurs significantly compared with other service recipient groups when adaptation is absent or modest. On the other hand, making an adjustment implies that we give more weight to the utility of people who do adapt compared with the utility of those who do not. Also, how might this adjustment be made? It is not necessarily clear, therefore, that having an instrument that measures (potentially) adapted utility is a limitation. Nonetheless, it would clearly help in resolving this issue if we knew more about the scale and consequences of adaptation.

The preference weights estimated from the BWS data have utilised the S-MNL model, and this has been used to incorporate both sample-level variance heterogeneity and preference heterogeneity. The focus for this research has been on specifying models that take account of observable heterogeneity, but there are additional models now available that allow specifications that also incorporate unobserved heterogeneity, for example the generalised MNL (G-MNL) model and scale-adjusted latent class model. The estimation of these models was outside of the scope of this study as they were still the subject of academic investigation at the award of the research grant, but they have now started to be applied in practice and could be utilised in this context. The data collected here could be used to support the exploration of these alternative models, with a view to understanding whether or not the relative value placed on preference weights of different SCRQoL states are sensitive to the incorporation of unobserved heterogeneity in the model specification.

Arguably, all valuation studies should be stratifying factors associated with heterogeneity and variation in preferences, in order to ensure that the end weights accurately reflect the preferences of the relevant population. While we did not find systematic differences between our service user and general population sample preferences, service users’ preferences were associated with their SCRQoL states. However, it is not feasible to establish the preferences of people in very poor SCRQoL states or people for whom the cause of impairment is associated with limited mental capacity, either in sufficient numbers or in some cases at all, for both practical and ethical reasons.

Implications and potential future uses

As we describe in Chapter 1, the project was conducted alongside other related work, which has allowed the measure with provisional preference weights to be made available prior to the end of the project (www.pssru.ac.uk/ASCOT). The measure has attracted widespread national and international interest. In England, the interest is owing in part to its inclusion in the annual national ASCS. Both the overall measure and individual items form part of the Adult Social Care Outcomes framework. 1 This means that a valuable source of benchmark comparative data will be available from 2011. However, using the measure for this purpose suggests that further work on establishing a means of adjustment to reflect SCRQoL ‘gain’ would be of particular value in order that data on SCRQoL ‘states’ are not misinterpreted.

At present there are no QALY equivalent measures available in social care. This study has attempted to fill this gap. A SCRQoL-based SC-QALY would have many uses, paralleling the use of these indicators in health care. First, it is hoped that a SC-QALY will become a routine measure in the economic evaluation of social care interventions. Predecessors of the SCRQoL have already been used in Department of Health-funded evaluations of individual budgets in social care, reablement services, and in the evaluation of personal health budgets. The anchoring of the SCRQoL to create a SC-QALY will allow generic incremental cost-effectiveness comparisons to be made between social care interventions, and between social care and health-care interventions.

Second, a SCRQoL measure is useful information in social care practice. For example, it offers a means for regulators to ensure that providers are delivering minimum standards of SCRQoL to service recipients. Commissioners could use QoL information in informing commissioning choices and even reimbursement levels for providers. Currently, social care services are allocated on the basis of need (i.e. the severity of a person’s condition and their non-service-related circumstances), but allocating resources on the basis of the potential marginal improvement in QoL offered by services is likely to lead to an overall improvement in productivity.

The study has identified that service user preferences are no different to those of the general population, but their experience of SCRQoL is. The study provides important insights into those preferences, supporting the policy emphasis on giving service users greater control. We have a basis on which to compare the SCRQoL of service users and the general population. Potentially, this provides us with a starting point for an ‘outcomes’ basis on which to debate and make difficult decisions about what we expect of social care services.

The anchoring exercise means that we now have estimated the necessary function for mapping BWS values onto TTO values. This means that, in theory, we can make direct comparison between ASCOT and EQ-5D values as they are on similar scales. Caution in interpretation will be needed as the upper anchor is different: EQ-5D is to ‘full health’, whereas ASCOT reflects the ‘ideal’ SCRQoL state. Nevertheless, ASCOT offers considerable potential to improve the sensitivity of economic evaluations to outcomes delivered, as health- and social care interventions are often closely linked, and to assist in making cost–utility comparisons across a wider range of intervention types than is currently possible.

Further work

While the measure provides an important step forward, much remains to be done. The interdependent nature of well-being of carers and those they care for means that at present the measure only partially reflects the impact of social care services. One preference-weighted measure has been developed,117–119 but ideally we would want a measure that reflects more closely the SCRQoL measured by ASCOT. Some work has already been conducted exploring the applicability of the domains and adaptations needed to reflect the equivalent of SCRQoL for carers. 120 Although the results of this work has been promising,121 there remain a number of challenges, not least establishing preference weights for this measure or a version of it, and identifying the best way to reflect trade-offs between the well-being of carers and cared-for individuals.

We have identified the limitations of applicability of the measure, as it stands, to those with cognitive impairment as the result of learning disabilities, dementia or other conditions. As part of the development work for the ASCS, a self-completion version for people with learning disabilities was developed, but it was acknowledged that this was not the ideal method for establishing SCRQoL for this group. 8 This was supported by a recent review of the ASCS. 7 A mixed-method approach for use in care homes has been developed. 8 However, this is resource intensive, and the high proportion of people with cognitive impairments of some sort who use adult social care services means further development work in this area would seem to be a priority.

While the findings are encouraging, evidence is needed about the reliability and sensitivity of the measure of current SCRQoL, facilitating correct interpretation of differences and change in people’s experiences: when is a difference ‘clinically significant’? Does context and type of impairment systematically affect responses?

As we note above, the use of ASCOT in the Adult Social Care Outcomes Framework (ASCOF) means that this will provide a valuable source of data, but if we are to reflect the ‘value added’ of services it is important that work is undertaken to aid interpretation. Further work building on the pragmatic approach reflected in the ‘expected’ and SCRQoL ‘gain’ measures could both feed into this and further test the validity and reliability of the approach.

Anchoring the measure to the ‘being dead’ state is only the first step if we are to be able to make best use of our SC-QALY alongside the long-established health care QALY. There are clearly relationships with EQ-5D, but many questions remain, not least of which is how our top anchor of ‘ideal’ SCRQoL state relates to ‘full health’.

In light of these questions and considerations, in terms of future research we would recommend as priorities for future work:

• extending the approach to include informal carers

• exploring the potential to link between ASCOT and EQ-5D in estimating and comparing QALYs

• developing validated approaches to establishing ASCOT states for service users living in their own homes who have cognitive impairment and communication difficulties

• further work to test the validity, reliability and sensitivity of the current expected and gain measures in a variety of contexts.

These future developments will extend the range of outcomes that we are able to reflect, and the applicability and value of the ASCOT measure itself. Although this list can be seen as a rank order of priorities, they are largely independent of each other. Developments associated with the ASCS may provide opportunities to develop methods for establishing ASCOT states for a wider range of groups and further test reliability and validity. We put the highest emphasis on the carer measure as this reflects an additional aspect of outcome that ASCOT does not currently encompass.

Conclusions

Resources are always limited, but this is even more the case in the current financial climate. This, together with the policy emphasis on outcomes, makes it all the more important that we have good measures to reflect the value of social care interventions for those who use them. In evaluating cost-effectiveness in health care, the QALY is widely accepted as a measure of outcome. Although related to health, social care has fundamentally different objectives, primarily compensating for rather than treating or mitigating impairment, meaning health QALYs will not accurately reflect the impact of social care services. The project was designed to establish the basis for an equivalent to the QALY for social care: a preference-weighted and ideally anchored generic measure of social care outcomes that could be used for a variety of purposes.

The first task was to clarify exactly what it is we are attempting to measure. Drawing on the SPOW and Sen’s concepts of functioning and capabilities, we built on previous work to design a theoretically grounded measure of SCRQoL. In addition to a measure of current SCRQoL, which can be used to reflect differences over time and between different populations, we developed a pragmatic approach to establishing the impact of services, which draws on the ongoing nature of most social care, which means people themselves daily face the possibility that the help and support on which they rely may not be forthcoming. This means it is feasible to ask people about their needs in the absence of the help provided, allowing us to derive an ‘expected’ measure: the difference between people’s current and expected SCRQoL reflecting the impact of services as SCRQoL ‘gain’.

The items were cognitively tested, and the measure tested for validity and feasibility with social care service users. Response rates were good for all items, which implied that both the current and expected SCRQoL items are feasible with this population. Despite concerns about the distributions of two items, which led to further development and testing, there was good evidence for the validity of the descriptive system, each item capturing a different aspect of SCRQoL. There was also evidence for the validity of the SCRQoL current and gain scales, which appeared to operate as intended. The final version of the ASCOT instrument is shown in Appendix 5.

We explored different techniques in establishing preference weights and anchoring the scale, breaking new ground in a number of areas. We tested empirically the comparability of the profile-case BWS and DCE estimates with the general population. The similarity of the results and comparative advantage of BWS in terms of smaller cognitive burden, while viewing all the attributes and allowing the estimation of a common scale, led us to use this technique for the main preference elicitation task. We tested the feasibility of identifying user preferences and found that once the introduction and layout were clear, service users generally were able to undertake the tasks. We established an approach for identifying people for whom the task was too cognitively demanding, and ways of identifying and investigating the incidence and impact of strategies, such as taking into account only certain domains. In addition, we tested and then implemented the TTO technique, the first time this had been used with respect to social care, in order to enable us to anchor the scale to ‘being dead’, widening the ways in which the measure could be used and enhancing comparability with health QALYs.

The preference study work involved a number of different data collections and techniques. The results showed a consistent picture in terms of both what we would hope to observe (higher values being put on higher SCRQoL states according to our definitions) and patterns of actual preferences. An important question was whether or not service users’ preferences differed from those of the general population. We found remarkably little difference. Indeed, the consistency in the estimates generated across techniques, time, the general population and service users gave us considerable confidence in our final utility weights.

While there is clearly room for further work, the study has made important progress in this neglected area. The revised ASCOT measure developed through the study shows considerable potential, providing a first estimate of a social care equivalent to the QALY, and which can be used in a range of circumstances, including cost-effectiveness, cost–utility and policy evaluation, to reflect the impact and value of social care interventions. However, if we are to fully reflect the outcomes of social care, an equivalent measure is needed for carers. If we are to exploit the full potential of the measure, it will be important to establish how SC-QALYs relate to health QALYs and, although we have provided a starting point, there remains much to do in developing an approach that comprehensively reflects outcomes for all service users and exploring and interpreting differences between groups and circumstances. Our ‘expected’ and ‘gain’ measures build on the nature of social care and represent a radically new approach to reflecting outcome. While this would benefit from further work, we see this as a pragmatic low-cost approach that shows considerable promise.

There has been widespread interest since ASCOT was first made widely available in June 2010 through www.pssru.ac.uk/ASCOT. The measure is already being incorporated in the annual national ASCS, and proposed as an important element of the Transparency in Social Outcomes framework, which should enhance its value by providing benchmark data about current SCRQoL states for a range of service user groups and contexts.

Contributions of authors

Ann Netten (Director of PSSRU and Professor of Social Welfare) was principal investigator and project manager.

Peter Burge (Associate Director, Choice Modelling and Valuation) led the work on developing the preference weights.

Juliette Malley (Research Fellow) led the work on developing the items and testing validity, and conducted analysis associated with these activities.

Dimitris Potoglou (Analyst, Choice Modelling and Valuation) contributed to the design and analysis of the preference experiments.

Ann-Marie Towers (Research Officer) undertook validity testing analyses.

John Brazier (Professor of Health Economics) advised on the development of the descriptive system and the design and analysis of the TTO valuation.

Terry Flynn (Head of Social Policy & Economic Evaluation) developed the theoretical and analytical BWS models used to value preferences.

Julien Forder (Professor of the Economics of Social Policy) advised on the design of the outcome measure and its potential application in social care.

Beryl Wall (Associate Director, Market Research) managed the data collection activity and advised on the presentation of the TTO interview.

Publication

1. Potoglou D, Burge P, Flynn T, Netten A, Malley J, Forder J, et al. Best-worst scaling vs. discrete choice experiments: an empirical comparison using social care data. Soc Sci Med 2011;72:1717–27.

Disclaimer

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

Data sets

Evidence for choice of domains

Food and drink

Personal cleanliness and comfort

Safety

Social participation

Control over daily living

Occupation

Accommodation cleanliness and comfort

Dignity