Improving the evaluation of therapeutic interventions in multiple sclerosis: the role of new psychometric methods

Article history

The research reported in this issue of the journal was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA Programme in April 2007 under the newly established NIHR Methodology Panel. The HTA Programme project number is 95/01/05. The contractual start date was in February 2005. The draft report began editorial review in January 2007 and was accepted for publication in March 2008. The commissioning brief was devised by the NCCRM who specified the research question and study design.

Declared competing interests of authors

None

Copyright statement

© 2009 Queen’s Printer and Controller of HMSO. This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NCCHTA, Alpha House, Enterprise Road, Southampton Science Park, Chilworth, Southampton SO16 7NS, UK.

2009 Queen’s Printer and Controller of HMSO

Overview

The aim of this chapter is to explain what is meant by traditional psychometric methods and to document their limitations. This will act as the basis for understanding the reasoning that led to the need for, and development of, new psychometric methods. We start the chapter by looking at two typical rating scales used in health measurement to explain what they are trying to achieve, how they ‘work’ and how they are most commonly evaluated. We then discuss the theory that underpins those methods and the limitations of that theory. Finally, we discuss the limitations of traditional psychometric methods for rating scale evaluation, development and handling of data.

Rating scales and how they work

Some things that we wish to quantify can be measured directly using devices or machines. Examples include weight, height and protein levels in the cerebrospinal fluid. Other things that we wish to measure cannot be measured directly. Examples include health variables such as disability, anxiety, fatigue and quality of life. These variables must be measured indirectly through their observable manifestations. For example, we can only measure the physical disability of a person by engaging the person in physical tasks.

In fact, all measurements, whether direct or indirect, are of this nature. The weight of a person can only be determined by engaging their weight with an instrument which reacts to it; the height of a person can only be measured by engaging it with an instrument against which we can read off its length. Likewise, we infer the extent to which a person is depressed from the symptoms of this health variable that the person manifests, either through observation or by formalising some questions. In order to stress this inferred aspect of health variables, they are typically referred to in the rating scale literature as latent traits.

Appendices 1 and 2 show two typical health rating scales, the Rivermead Mobility Index (RMI)1 and the Multiple Sclerosis Impact Scale (MSIS-29). 2

Consider first the RMI (see Appendix 1). This is a rating scale purporting to measure mobility. It has 15 questions (items). Each item concerns a mobility-related task that has two response categories: ‘no’ – I am unable to do this task; ‘yes’ – I am able to do this task. The RMI is scored by clinicians from patient interview and/or observation. Scores for people are constructed by counting (summing) the number of ‘yes’ responses. Higher scores indicate greater ability and less disability. By convention, the scale scores achieved by summing item scores in this way are called total scores, raw scores or summed scores. These three terms are often used interchangeably.

Items with two response categories are called dichotomous items. It is more common for the items of health measures to have three or more response categories that are ordered from less to more (or vice versa). These are called polytomous items. An example is the MSIS-29 (see Appendix 2.1), which purports to measure the physical and psychological impact of multiple sclerosis (MS). Each item has five response categories. However, the same scoring process applies. Sequential integers (1, 2, 3, 4 or 5) are assigned to the ordered response categories and scores for people are constructed by counting the integer responses across the items. The response categories and their integer scoring system are sometimes called the item scoring function.

The purpose of most health rating scales is to measure people. More correctly, they measure an aspect of people, such as their mobility (RMI) or the physical and psychological impact of MS (MSIS-29). In the light of this fact, consider the aims of the RMI and the MSIS-29. For the RMI mobility is being thought of (conceptualised) as a quantitative variable in the sense that it reflects a property that can have a range of values from ‘less’ to ‘more’. The 15 RMI items attempt to map out this idea so that responses to the 15 RMI items can be seen as indicators of the level of mobility. Essentially, the RMI seeks to map out mobility as a line (continuum) varying from more to less on which people can be located. Their location is determined by their total score. Thus, the 15 RMI items make operational (operationalise) the idea of the mobility variable. Because mobility is observed by a variety of manifestations, rather than directly, it is considered to be a latent (hidden) property. The words ‘trait’ , ‘construct’ and ‘aspect’ are typically used instead of the word property.

Figure 1 represents graphically, and simply, this conceptualisation of the mobility variable by the RMI, and the idea of measurement by locating a person on that line mapped out by the 15 items of the RMI. The notion of locating people (or something) on a line (or continuum) is an important one, is common to all measurement and is often termed ‘scaling’ .

FIGURE 1.

Conceptualisation of the Rivermead Mobility Index. This shows that a variable, here mobility, can be represented as a line, or continuum, ranging from less mobility to more mobility and also illustrates the location of a person on the variable and indicates the amount of mobility that person has.

This conceptualisation implies that each item represents a mark on the ‘ruler’ of mobility mapped out by the RMI. More specifically, the mark defined by each item represents the transition point of the score from 0 to 1, i.e. the point at which a person moves from scoring ‘0’ (unable to do) to ‘1’ (able to do), and the point below which he or she scores ‘0’ and above which he or she scores ‘1’ .

The MSIS-29 is slightly more complex. First, the RMI generates one total score whilst the MSIS-29 generates two total scores. Items 1–20 are summed to generate the total score for the physical impact subscale; items 21–29 are summed to generate the total score for the psychological impact subscale. Second, each item provides multiple (four) marks on the continuum, because each mark represents the transition between two adjacent response categories (i.e. ‘not at all’ /‘ a little’ ; ‘a little’ /‘ moderately’ ; ‘moderately’ /‘ quite a bit’ ; ‘quite a bit’ /‘ extremely’). Finally, note that the ordering implied by the response categories goes in different directions. For the RMI high scores indicate more ability; for the MSIS-29 high scores indicate more disability. This simply means that the continua implied by the two scales run in different directions.

Psychometrics is defined as the study of methods for measuring psychological variables. The field has broadened to include many circumstances where rating scales are used as measurement instruments. Essentially, when evaluating a scale such as the RMI, the aim of a psychometric analysis is to determine whether this idea (conceptualisation) of mobility as a variable mapped out by the 15 items of the RMI has been achieved. In measurement speak, the purpose of a psychometric analysis is to establish the extent to which a quantitative conceptualisation has been operationalised successfully. 3 To achieve that goal a range of evidence is used.

Traditional psychometric methods for evaluating scales

There are many different psychometric methods. This monograph concerns three: traditional psychometric methods, Rasch analysis and Item Response Theory (IRT). Other methods include Thurstone’ s method of paired comparisons,4 Thurstone and Chave’ s method of equal-appearing intervals,5 Likert’ s method of summated ratings6 and Guttman’ s scalogram analysis. 7 There are many more. Edwards’ text8 gives an excellent account of some of the earlier methods.

Different psychometric methods use different ranges of evidence to determine the extent to which a quantitative conceptualisation has been operationalised successfully. Traditional psychometric methods, the commonest psychometric method for evaluating rating scales for nearly a century, use evidence predominantly from correlations and descriptive statistics.

Typically, traditional psychometric methods consider evaluating rating scales in terms of three main properties: reliability, validity and responsiveness. However, there is no consensus in the literature, although there are guidelines9 as to how the results of psychometric analyses should be reported. Hence, they are documented in the literature in a variety of ways. We have previously recommended that a comprehensive scale evaluation using traditional psychometric methods should involve the examination of six psychometric properties: data quality, scaling assumptions, targeting, reliability, validity and responsiveness. 2 Data quality concerns the extent to which a scale can be administered successfully in the target sample. 10 Tests of scaling assumptions examine whether it is legitimate to sum item scores to generate a single scale score. 11 Targeting concerns the extent to which the distribution of disability in the sample matches the range of disability measured by the scale. 12 Reliability describes the extent to which scale scores are free from random error. 13 Validity refers to the extent to which a scale measures what it purports to measure. 13 Responsiveness is the ability of an instrument to detect change accurately when it has occurred. 14 These methods are fully documented elsewhere in a manner accessible for clinicians2,12,14–16 and are described more fully in the Chapter 4, which documents the evaluation of the RMI.

Classical Test Theory: the theory underpinning traditional psychometric methods

Traditional psychometric methods are underpinned by a theory called Classical Test Theory. It has also been termed Classical True Score Theory, Weak True Score Theory17 or the Classical Test Model. 18 A test theory, or test model, is a mathematical representation of the factors influencing scores generated by a rating scale, and is described by its assumptions. 19 The reason that the word ‘test’ (which often confuses clinicians) is used is that the early work concerning the use of rating scales as measurement instruments was in the fields of education and psychology where the aim of measurement was often testing.

Classical Test Theory is a theory of how errors of measurement can influence the scores achieved on rating scales. The theory assumes certain conditions to be true. If these assumptions are reasonable, then the conclusions derived from the model are reasonable. If the assumptions are not reasonable then use of the model can lead to faulty conclusions. Classical Test Theory has seven main and fairly straightforward assumptions. It is interesting to note that a few simple assumptions expand into a set of principles for developing rating scales and for testing their reliability and validity. 19

Limitations of traditional psychometric methods

Irrespective of these concerns about the theory underpinning Classical Test Theory, there are significant problems with traditional psychometric methods of evaluating scales, handling their data and constructing scales. The most clinically important of these are discussed below.

Summary

Health rating scales are measurement instruments whose aim is usually to quantify characteristics of people that cannot be measured directly. They are constructed to map out variables as a line on which people can be located. Psychometric methods concern the construction and evaluation of rating scales in order to determine their success at making the variable operational and locating people.

The most widely used method for constructing and evaluating rating scales is termed traditional psychometric methods. Underpinning these methods is Classical Test Theory, which postulates the existence of a true score, that error scores are uncorrelated with each other and with the true scores, and that observed true and error scores are linearly related. However, because true scores and error scores cannot be determined, the appropriateness of the assumptions cannot be verified and we can only surmise that they are met.

Most health rating scales use Likert’ s method of summated ratings. Here, sequentially ordered response options are allocated sequentially ordered integers, and item scores are summed to give total scores. Whilst this method is nearly ubiquitous in health measurement, the belief that ordinal-level total scores approximate interval-level measurements is not well founded. Other problems arising from the use of total scores and traditional psychometric methods are that evaluations of scales are sample dependent and the measurement of people is scale dependent. Even at a purely logical level these undermine a belief in the interpretation of total scores as measurements.

Overview

Chapter 1 identified that most rating scales in health care use Likert’ s method of summated ratings as their format, are constructed and evaluated using traditional psychometric methods (if any psychometric methods are used) which are underpinned by Classical Test Theory and use total scores as the basis for their analyses. It then highlighted the limitations of these approaches, specifically that the ordinal total scores generated by Likert’ s method of summated rating are not interval-level measurements, that the theory of Classical Test Theory cannot be verified in data and that the analysis of total scores as measurements is problematic.

This chapter concerns developments that have been undertaken to overcome the limitations of Classical Test Theory, traditional psychometric methods and total score analysis. The main outcomes of these developments are two ‘new’ psychometric methods: Item Response Theory (IRT) and Rasch measurement. Essentially, like Classical Test Theory, the new psychometric methods postulate theories of how the scores generated by rating scales relate to measurements of the variables they seek to estimate. However, unlike Classical Test Theory, they provide mathematical realisations of these theories (mathematical models) that enable their verification in data. That is, they can be formally and rigorously tested.

This chapter charts the development of the two new psychometric methods, highlighting that they come from different origins and represent different research perspectives, despite clear similarities. Perhaps not surprisingly, this has led to confusion and misunderstandings about their similarities and differences. We explore why this might have occurred.

The fact that IRT and Rasch measurement are fundamentally different means that two groups have evolved: proponents of IRT and proponents of Rasch measurement. We examine why each group has chosen to adopt their approach and not the other. However, because of the similarities between IRT and Rasch measurement, and the resultant confusion that has arisen, it is our experience that there is a third group: those who do not know that there is a difference between the two.

Background to the new psychometric methods

The term ‘new’ , or ‘modern’ , psychometric methods refers, mainly, to two methods for constructing and evaluating rating scales, or analysing rating scale data. These methods are called Item Response Theory (IRT) and Rasch measurement. The new psychometric methods represent a logical progression from Classical Test Theory, because they attempt to improve the scientific quality of the theory underpinning rating scales. For Classical Test Theory the underpinning theory is weak, and by definition the conclusions about the performance of scales as measurement instruments and the measurement of people cannot be strong. 19 The logical next step was to try to strengthen that theory and, therefore, the scientific quality of the conclusions that could be drawn.

A theory is basically an idea about something. In medicine, theories are proposed about the causes of disease and evidence is gathered which either supports or refutes the theory. In this way theories become refined and developed. Eventually, usually aided by these theories and a bit of serendipity, the cause of a disease is often discovered, and we move on. However, this is not always the case; for example, although various theories have been proposed, the cause of multiple sclerosis remains elusive.

Other theories are ideas about the relationship between variables and entities. These theories serve to describe, explain or predict some behaviour or occurrence. Structure is brought to theories by postulating formal models. These are explications of theories, or parts of theories, that connect the theory to observable events. Their function is to permit logical deduction of relationships that have not been demonstrated but may be demonstrable.

One method of bringing a concrete structure, or framework, to a theory is to express it as formal logical (mathematical) relations between variables or events. The articulation of a theory as a mathematical model (i.e. equation or formula) has three main advantages. First, because equations predict the relationships between variables, they enable careful and sophisticated checks on the consistency of the data to determine how observations (observed data) satisfy the ‘fit’ of the predictions of a model, i.e. is the extent to which the data satisfy the theory, or alternatively the extent to which the theory is supported by the data. Second, they enable predictions to be made for the future.

The third value of mathematical models comes from the analysis of deviations of the observed data from the predictions of the mathematical model. These analyses can lead to important developments because they allow theories and models to be explored in two directions. One direction is to question whether the model is an adequate representation of the theory. This can lead to developments in our understandings of the theory and changes in the models used to formalise them. However, this direction requires confidence in the observed data. The other direction in exploring discrepancies between observed data and the model prediction is to question whether the data are adequate. This can also lead to developments in our understandings of the theory, but is more likely to lead to developments in understandings of the data themselves and how they might be collected. This direction requires confidence that the models used are adequate representations of the theory.

If sets of items are to be used as a measurement instrument, a theory is needed to underpin the development of scales, evaluation of scales and analysis of rating scale data. Otherwise any collection of items could be used in the belief that they represent a measurement instrument. We outlined earlier that Classical Test Theory is a theory of measurement error. To recap, it postulates that, when using a rating scale, a person has a ‘true score’ . A person’ s true score is the score a person would get on a rating scale if there were no measurement error. Thus, it is unobservable because of measurement error. The only observable measurement is the actual score on the scale – the ‘observed score’ . The theory then postulates that there is an additive relationship between these three scores (true = error + observed) and expresses this relationship as a mathematical formula (model or equation):

The theory also postulates that the size of the error is not correlated with the size of either the true or the observed score, and that the size of the error on one scale is not correlated with the true or observed scores on another scale. Finally, the theory postulates that the true, observed and error scores are linearly related, i.e. they have a straight line relationship. Thus, a fixed unit of change in one scale means a fixed unit of change in the other. It does not mean, necessarily, that the unit of change is the same. We have seen that from these statements postulated by Classical Test Theory, and its mathematical model, we are able to draw a number of conclusions and derive further equations for testing scale reliability and validity.

The main problem with Classical Test Theory, as we have seen, is that it is a weak theory. This is because the postulated statements constituting the theory, and the derived mathematical equations, could not be tested, verified or refuted. Therefore, it is an easy theory to satisfy. The new psychometric methods are an attempt to raise the scientific bar of the theories underpinning rating scales. Like Classical Test Theory, the new psychometric methods concern the relationship between a person’ s unobservable true measurement and his or her observed score. Unlike Classical Test Theory, the new psychometric methods focus on the relationship between a person’ s unobservable measurement on the underlying trait and the probability of responding to one of the response categories of a scale item.

There are a number of points to note about the final sentence of the last paragraph. The term ‘person’ s unobservable measurement’ is used because the new psychometric methods recognise the limitations of total scores. The new methods are interested in someone’ s true measurement on the construct being measured by the scale, i.e. his or her location on an interval-level continuum, rather than the true total score, which we know is ordinal in nature. The second point to note is that a person’ s true interval-level location governs his or her response to an item, i.e. the location on the construct should determine which item response category a person chooses on an item. This is logical. We would expect a person who is severely disabled to be unable to do many tasks of the Rivermead Mobility Index (RMI). We expect such a person to answer ‘no’ to most items. Thus, the response to any item is a function of someone’ s level of disability. Also logically, a person’ s response to an item is related to the difficulty of the tasks: running (RMI item 15) is usually a more difficult task than turning over in bed (RMI item 1) if a person is physically disabled. The third point to note is the use of the phrase ‘probability of responding to one of the response categories of a scale item’ . This recognises that it is unreasonable to try to predict exactly how someone will respond to an item. We can only say that he or she is likely to choose a specific response category. Thus, the relationship between true interval-level measurement and response to an item is best formalised in terms of probabilities. The final point to note is that the focus has changed from the total score level in Classical Test Theory to the item score level in the new psychometric methods.

It seemed reasonable, therefore, to consider developing mathematical models that would relate the probability of a response to an item to the person’ s location on the continuum measured by the items and to some characteristics (parameters) of the items. The challenge then was to find, or derive, a mathematical model that would link these variables together, and give the information required, using the responses of a sample of people to a set of items. Essentially, the mathematical model must use this information to allow us to determine if a set of items forms a reliable and valid rating scale on which people can be located, preferably in interval-level units.

At this stage it is worth noting some issues relating to terminology. The term ‘Item Response Theory’ is used, not surprisingly, because both of the new psychometric methods concern theories about responses to items. Confusion has arisen because some people use the term to encompass both branches of new psychometric methods (IRT and Rasch measurement), while others use it to mean only the IRT branch of new psychometric methods. Other synonyms for the new psychometric methods are Person–Item Response Theory, Strong True Score Theory (to differentiate from Weak True Score Theory), Latent Trait Theory (to emphasise the fact that the constructs being measured are unobservable or latent) and modern test theory.

As pointed out in the introduction to this chapter, the two new psychometric methods have different origins and directions and represent different perspectives. Thus, they are fundamentally different. However, to complicate matters and ensure confusion, they also have important similarities.

Development of Rasch measurement

At the same time that Frederic Lord was developing his theories at the Educational Testing Service in the US, Georg Rasch was developing his theories at the Danish Institute of Educational Research in Copenhagen. 65 It is important to note that Rasch’ s work was independent of Lord’ s (and vice versa), although Lord discusses Rasch’ s work in his landmark text published 8 years later. 17 In addition, to our knowledge, Rasch’ s work was independent of influence from any other work into psychometric methods because, it is said, he rarely read research papers.

At the beginning, Rasch took a similar approach to Lord’ s in that he tried to model the data. He was working with reading test data, in particular students’ ability to read texts of varying difficulty, and the distribution of errors. He chose to work with a Poisson model as this is generally used as an error count model. However, to make the Poisson ‘work’ he needed to have a parameter for each person’ s ability and each reading text difficulty. Thus, Rasch treated each student in his data set as an individual. This contrasts with more conventional mathematical modelling in which people tend to treat groups of individuals as random replications of each other so that the error includes the variation among individuals. He fixed those, and there was randomness only in the response process of a person to an individual text.

This work gave Rasch a very elegant model in which he could eliminate the person parameter to carry out tests of fit and to get the relative difficulty of the texts. He showed this to Ragmar Frisch (Norwegian economist and Nobel Prize winner), who noted that the person parameter dropped out. This observation made Rasch think more closely about the mathematics of the model and the class of all such models. From this he derived the specific model for dichotomous responses independently of any data of the kind. He then analysed two existing tests according to this model. One set of data fitted the model, the other did not. 66,67

Over time, Rasch increasingly noticed the implications of a property of the model he had deduced. The property was that the item locations and person locations could be estimated independently of each other. This will be discussed and demonstrated later. However, it is the implication of this property that is important. It means that the measurement of people can be made independently of the sampling distribution of the items used, and the location of items on the continuum can be made independently of the sampling distribution of the people from whom they are derived. Put another way, this means that the relative locations of any two people does not depend on the items they take, and that the relative locations of any two items does not depend on the people from whom the estimates are made. This is Rasch’ s criterion of invariance – or stability. 66

Of course there is one essential proviso: that the data fit the Rasch model. This is because the property of invariance is a consequence of the model, not of the data or just the application of the model. Thus, for invariance to be a consequence of the data, the data must fit the Rasch model within statistical reason. However, this means that when the data do fit the Rasch model, different subsets of items will give equivalent person estimates and different subsets of persons will give equivalent item estimates.

Rasch realised that this property was a fundamental property of a very important class of models. From then on he shifted his focus from describing data sets to studying a class of models with a unique property. The implications are discussed further below.

Rasch developed his model for dichotomous variables. His work was developed at the University of Chicago by Wright,68 with Stone69 and Masters,39 and by Andrich, who generalised the Rasch model for use in rating scales with polytomous response categories. 70

In the Rasch paradigm, the mathematical model is given primacy. This is not because it describes any set of data. The case rests on the inherent properties of the model. Essentially, it provides the optimum criterion for fundamental measurement.

The general approach in Rasch measurement is to use only Rasch models to construct scales, evaluate scales and analyse rating scale data. The use of other item response models is not typically considered. When the data do not fit the Rasch model, another model is not chosen. Instead, the finding invokes an examination of the data to determine why, for example, a set of items hypothesised to be a measurement instrument are not performing as such. Thus, the justification for model selection is theoretical evidence of its suitability. 23 The data are not considered given. It is important to note that this is not the typical approach to using mathematical models with data. In addition, the finding that observed data do not fit the chosen model means that developments in construct theory, and the items used to operationalise them, is justified empirically.

Item Response Theory and Rasch measurement: similarities and differences

Similarities

There are a number of important similarities between IRT and Rasch measurement. This only serves to increase the potential for confusion. The first similarity is that both families are item response models for constructing and evaluating rating scales and analysing their data. The second similarity is that both appeared at around the same time. The first written reports about both IRT and Rasch measurement were written in the 1950s71 and their landmark texts were published in the 1960s. 17,65

Perhaps the greatest similarity between IRT and Rasch measurement is the structure of the mathematical models. 33 Figure 2 shows the mathematical models of the Rasch model and the two-parameter and three-parameter models. All three are ‘logistic’ models. This simply means that central to them is the expression (e/1 + e). Hence, the Rasch model is known as the simple logistic model (SLM) and the two IRT models as the two- and three-parameter logistic models (2-PL, 3-PL). Second, the models are such that:

• if C = 0, the 3-PL becomes the 2-PL

• if a = 1, the 2-PL becomes the SLM

• if a = 1 and C = 0, the 3-PL becomes the SLM

where C = constant, describing the lower asymptote due to guessing and a = discrimination.

FIGURE 2.

Formulae for the Rasch model, and for the one- two- and three-parameter logistic models. c_i = constant, describing the lower asymptote due to guessing for item i; a_i = slope of item i (discrimination); D_i = difficulty of item i; and the one-person parameter is B_n = ability of person n. Note. Some refer to the Rasch model as the ‘one-parameter model’. This is the case here to demonstrate the mathematical similarity of the models. However, as explained in the text, the conceptions of the Rasch model and the 1-PL IRT model were very different.

This has led many people to consider the Rasch model as nothing more than a ‘special case’ of the 2- and 3-PL models with convenient properties,72,73 and to describe it as a one-parameter logistic model (1-PL). That perspective and terminology is reasonable if the aim is to find the model that best fits the data. However, it ignores the fundamental reasoning behind the development of the Rasch model and its value. That perspective is not reasonable if the reasoning behind the development of the Rasch model is taken into account. This is because the addition of parameters to the Rasch SLM prevents the separate estimation of item and person parameters and thus the property of invariance. Massof33 demonstrates this formally.

There are two more similarities between IRT and Rasch measurement that are noteworthy. The first is that proponents of both have some common goals despite their different agendas. Both approaches are concerned with the development of better measurement methods, improved evaluation of scales and greater understanding of the constructs that we seek to measure. 64,74 The second is that both approaches often acknowledge the importance of Thurstone’ s work, but for different reasons. Proponents of IRT acknowledge Thurstone for being the first person to apply mathematical models to the field of ‘social’ measurement. Thus, IRT is in keeping with their approach. Proponents of Rasch measurement acknowledge Thurstone’ s work because he laid down a series of requirements for measurement, which he argued rating scales should satisfy to be considered valid (and which Likert misinterpreted as assumptions75). Thus, Thurstone is advocating the primacy of the theory and the model.

Why proponents of Item Response Theory favour their approach

Any opinion necessarily consists of arguments for it and counter-opinions against it. Proponents of IRT give primacy to the data and an attempt to best explain it. Thus, it is easy to understand why one might seek to find the item response model with the best fit. Another reason people favour the IRT approach is concern about using an item response model that does not include an estimate of item discrimination, given that there is empirical evidence to demonstrate that items have different discriminatory abilities. 76 A further reason supporting the IRT approach is concern about the concept of finding data to fit the model. This has been felt by some to disturb construct validity. 76 However, giving primacy to the data does imply confidence that the data are not fallible. This might raise cause for concern given the ambiguities of items and the fact that many items could be placed in scales measuring a range of constructs.

Why proponents of Rasch measurement favour their approach

The reason that proponents of Rasch measurement favour their approach, and give primacy to the choice of model, lies in the inherent properties of the model. In essence, it offers the optimum criterion for fundamental measurement. 66 It is appropriate to explore this statement further because it goes right back to the meaning of measurement itself.

Many psychometric textbooks quote the definition of measurement formalised by Stanley Smith Stevens,77 but which Stevens,77 and others,78,79 attribute to Campbell. 80 He defined measurement as ‘the assigning of numbers to things according to rules’ . Stevens proceeded not to define the rules for assigning numbers to things but rather to offer a four-level classification of scales (nominal, ordinal, interval, ratio) and the statistical tests that ought, or ought not, to be applied to them. Although there have been other variants (and combinations) of these, Stevens’ classification of scales as ordinal, interval and ratio has been widely adopted as the basic scales of measurement and has had a major influence on psychometric theory and its development.

It is perhaps interesting to note that many standard psychometric texts over the years19,79,81–88 offer little discussion of the topic of the nature of measurement, despite the tomes devoted to it. 81 In fact, measurement is a very important and advanced concept. 66 It has been defined as the sine qua non of science,89 Helmholtz90 is reputed to have stated that all science is measurement,and it is widely acknowledged that advances in measurement methods have underpinned the progress of physical sciences. 66 Measurement in the physical sciences, which was called fundamental measurement by Campbell in the 1920s,91 has been studied and debated at length by mathematicians, philosophers and social scientists in order to articulate the characteristics of measurement, against which systems purporting to generate measurements (for example, rating scales) can be tested.

Underpinning most physical measurement systems, e.g. weight and height, there is an empirical combining (concatenating) process. As Perline et al. 92 state: ‘It is easy to show that two lumps of clay joined into one is equal to the sum of the weights of the individual lumps.’ Campbell91 deduced that the ability to concatenate was the fundamental property on which physical measurement (i.e. measurement in physics) was based, coined the term ‘fundamental measurement’ and argued that there could be no fundamental measures in psychology because concatenation of psychological properties seemed impossible.

This challenge motivated the theory of conjoint measurement, which aimed to determine the conditions required for rating scales to achieve fundamental measurement. Luce and Tukey93 made an important contribution to the field in 1964.Essentially, for an item response model to produce fundamental measurements it must satisfy the mathematical requirements of a non-interactive conjoint structure. 94 These mathematical requirements concern relationships between the three key variables in the model: the person location and item location estimated by the model and the observed responses to items. A non-interactive conjoint structure is one that exhibits additive relationships, double cancellation, solvability, the Archimedean axiom and the independent effects that the item locations and person locations have on the observed responses.

The Rasch model satisfies these five conditions. Other item response models do not. 66 This underpins why the Rasch model offers the optimum potential for constructing fundamental measurements from rating scale data,66 why the status of other item response models as measurement models has been challenged33 and why proponents of Rasch measurement favour their models and do not tend to use others.

Limitations of new psychometric methods

The main limitation of the new psychometric methods is that they require a complex, more advanced level of mathematical understanding, and investment in training in the use of new software. For these reasons, Rasch analysis appears complicated and is not widely used, and there are few clinicians and researchers trained in its use and interpretation. Thus, the key question is ‘Do the clinical advantages of Rasch analysis outweigh the necessity for specialised knowledge and software?’ We believe that the new methods offer clinically meaningful, scientific advantages that far outweigh concerns about the necessity for specialised knowledge and software. In particular, the benefits of Rasch analysis provide a substantial leap from traditional methods in measuring patient outcomes. These benefits are that Rasch analysis: (1) offers the ability to construct interval-level measurements from ordinal-level rating scale data, thereby addressing a major concern of using rating scales as outcome measures; (2) enables us to obtain estimates suitable for individual person analyses rather than only for group comparison studies; (3) enables us to use subsets of items from each subscale rather than all items from the scale, yet still allows us to compare scores using different sets of items; and (4) allows for missing item data to be handled scientifically, rather than on the basis of assumption, because Rasch analysis computes an estimate from the available data rather than requiring missing data to be imputed.

We hope that this monograph will help to illustrate some of the reasons behind our belief in the relative advantages of Rasch measurement.

Summary

The aim of this chapter was to try to introduce the new psychometric methods – IRT and Rasch measurement. Yet, they are not so new; both approaches were available 40 years ago. However, they are complicated and there are few non-technical accounts that are accessible to people from a clinical background. To that end, we have tried to provide some insights to help interested people understand some of the fundamental issues and enable them to access what is undoubtedly a complicated literature.

Classical Test Theory is weak and the field of measurement using rating scales needed strong theories to underpin the science for it to be taken seriously. Two independent fields evolved, but they had many similarities that were bound to cause confusion, especially when coupled with some non-specific terminology.

The fundamental difference between Item Response Theory and Rasch measurement is substantial but subtle: one gives primacy to the data, the other gives primacy to the mathematical model. This results in a different approach to addressing measurement problems, and different approaches to the management of the findings. It is hardly surprising that Andrich66 argues that they are incompatible paradigms with proponents who are most likely to agree to disagree. There have been many misunderstandings between the two fields in the past, but it is interesting to note how infrequently the true differences of the two approaches are acknowledged publicly. As a consequence, our experience that few people are aware of the differences and similarities comes as no surprise.

The purpose of attempting to compare and contrast the two approaches is to help health-care professionals make up their own minds as to which they find most suited to their needs. In this chapter we have not argued that one way is right and the other is wrong. Certainly, both are superior to Classical Test Theory from a scientific perspective. We are proponents of the Rasch approach, the reason for our perspective being the opportunities offered by the Rasch model for advancing measurement in health care and, as a consequence, improving patient care.

Overview

In Chapter 2, we discussed that the need for strong theories, articulated in the form of mathematical models, underpinned the new psychometric methods. We identified that there were two main directions within the new psychometric methods – Item Response Theory and Rasch measurement – and that these methods had fundamental differences.

This chapter, and the rest of the monograph, focuses on Rasch measurement. We start by discussing the theory behind the Rasch model and showing how it is derived. Next, we explain and demonstrate two important properties inherent to the mathematics of the Rasch model: the fact that item and person estimates can be generated independently of the sample distributions of each other; and the fact that total scores for persons and items are ‘sufficient statistics’ for locating items and people on the continuum. Finally, we try to bring it all together and explain how a set of ‘yes’ /‘ no’ responses of a sample of people to a set of items can be used to generate estimates of items and persons that, when the data fit the Rasch model, are stable linear estimates. In this chapter we also need to introduce some of the mathematics.

Theory

There are two main components to the theory of Rasch measurement. The first component is that a person’ s response to an item is governed by only two factors: the location of the person and the location of the item on the shared continuum. The second component is Rasch’ s criterion of invariance, specifically that the relative location of any two persons on the continuum should be independent of the items used to make that comparison. The symmetrical aspect of this criterion is that the relative location of any two items on the continuum should be independent of the persons used to make that comparison.

It is valuable to recap, and develop, some of the issues. When a set of items is used as a measurement instrument, the aim of the items is to map out a continuum onto which people can be measured (located). It is more correct, as Thurstone31 pointed out, to say ‘on which aspects of people’ can be located. This is because we measure something about people, for example their height, weight, mobility, disability or quality of life. As the items mark out the measurement continuum, their locations define it. Thus, there is a common continuum on which the aspect of the people that is to be measured and the items for doing the measuring are located. Thus, when we use a set of items to measure people there is a simultaneous interaction between people and items.

A simply analogy may help to make this idea of simultaneous interactions concrete. This analogy concerns the measurement of strength and it seems to have been used first by Edwards8 in explaining Thurstone’ s method of paired comparisons. A similar analogy was used to illustrate the Rasch measurement model by Wright in his lectures, and by Wright and Stone in their publications. 95,96

Let us imagine that an investigator wants to determine the weights of a set of objects which range from very light to very heavy. Unfortunately, he does not have a physical scale for measuring weight. One alternative is to present the objects to a sample of people and ask them to make a judgement about the weights. For example, people could be asked to arrange the objects in order from lightest to heaviest, or they could be presented with all possible pairs of objects and asked to judge which object in each pair was heaviest. In fact, these two approaches underpinned Thurstone’ s method of equal-appearing intervals5 and the method of paired comparison. 4 Another approach is simply to ask each person to lift each object and record whether he or she can or cannot lift it.

The analogy of people lifting objects highlights that every observation involves a simultaneous interaction of multiple forces. 96 Here, the simultaneous interaction is between the person’ s strength and the object’ s weight, and the shared variable is of strength/weight. However, does a person lift an object because that object is light or because the person is strong? Similarly, does a person fail to lift an object because that object is heavy or the person is weak? Clearly, any interpretations of these observations will be confounded until there is a way of making separate estimates of the different forces (here, person strength and object weight). This, as we have seen, is a problem with traditional psychometric methods – the measurement of people depends on the scale, and the properties of the scale depend on the sample. They cannot be separated.

Rasch’ s criterion of invariance warrants a further comment. When two people are measured they are located on the continuum of whatever is being measured, e.g. mobility using the RMI. Rasch’ s criterion of invariance is that the relative locations of these two people should be the same regardless of the RMI items chosen to measure their mobility. Similarly, the relative location of any two items on the continuum should be the same regardless of the people chosen to generate those estimates. Note that this refers to the relative locations, not the absolute locations of persons and items. The relative locations refer to the distances between the locations of people or items.

Mathematics

We will start by pursuing the people lifting objects analogy. Logically, when any person in the study sample (standard notation = n) attempts to lift any of the study objects (standard notation = i), the thing that governs the likelihood (probability) of a successful lift is the difference between a person’ s strength (standard notation = B) and the object’ s weight (standard notation = D). Thus, for person n lifting object i, the probability of that person lifting that object is governed by the difference between the strength of person n (B_n) and the weight of item i (D_i), i.e. (B_n – D_i).

When person n is stronger than an object i is heavy, this difference is greater than zero (B_n – D_i > 0), and thus the probability of a successful lift is more than 50% (> 0.5); and the more B_n exceeds D_i,the greater the probability of a lift. Similarly, when person n is weaker than an object i is heavy this difference is less than zero (B_n – D_i < 0), and the probability of a lift is less than 50% (< 0.5); and the more D_i exceeds B_n the greater the probability of failing to lift. Logically, therefore, when a person is as strong as an object is heavy, the strength/weight difference is zero (B_n – D_i = 0), and the probability of a lift is 50% (= 0.5).

Now consider this logic in relation to any person with MS from the CAMS study completing the RMI. The actual response to any of the 15 items is determined by the person’ s mobility level and the item’ s difficulty. Also, the probability that the person will choose one of the two response categories (‘No, I can’ t do this task’ or ‘Yes, I can do this task’) is governed by the difference between the his or her location on the mobility continuum and the item’ s location on the same continuum. Thus, when person n, whose location is B_n, answers RMI item i, whose location is D_i, the probability of responding ‘yes’ or ‘no’ is governed by the value (B_n – D_i).

The general relationship between the probability of any person responding ‘yes’ to any item of the RMI, and the difference between that person’ s and the item’ s locations (B_n – D_i) on the mobility variable can be shown graphically (Figure 3a). The resultant curve is S-shaped (also known as an ogive). Three things about this curve are worth noting:

1. The probability of responding ‘yes’ (y-axis) never reaches 0 or 1 – as there is always a chance, however small, of a person being unable to do a task. That is the nature of probabilities.

2. The difference between B_n and D_i on the mobility variable (x-axis) ranges from minus to plus infinity (–∞ to + ∞), i.e. the range is unbounded.

3. The graph relating the probability of responding ‘no’ to any item and the difference (B_n – D_i) on the mobility variable is the reciprocal curve, and is shown in Figure 3b.

FIGURE 3.

(a) Probability of responding ‘yes’ to an item versus difference between person and item locations. (b) Probability of responding ‘no’ to an item versus difference between person and item locations. (c) Probability of responding ‘yes’ or ‘no’ to an item versus difference between person and item locations.

When Figure 3a and b are combined to give Figure 3c we have the type of graph that enables us to predict, for any difference between person location and item location on the mobility variable, the probability of responding ‘yes’ and ‘no’ . From Figure 3c note that:

• The curves cross at probability = 0.5 (y-axis) and B_n – D_i = 0 (x-axis).

• The sum of the probabilities of responding ‘yes’ and ‘no’ at any point on the x-axis is 1.0.

• These graphs tell us neither how able an individual is nor how difficult an item is – they merely give the probabilities of responding ‘yes’ or ‘no’ governed by the difference between person mobility and item difficulty.

So far we have postulated a theory that the difference between the person’ s location and the item’ s location on the mobility continuum governs the probability of the outcome of any person’ s response to any item. This theoretical relationship for the ‘yes’ response is an ogive (see Figure 3a). If this theory can be articulated as a formula, we would have a mathematical model relating the probability of a ‘yes’ response to the difference between the person location and the item location on the shared mobility continuum. If such a formula existed we might be able to deduce the mobility of people and the difficulty of the RMI items from a matrix of responses of people to the RMI (how is explained later).

Essentially, we need to be able to relate bounded probabilities (0 ≤p ≤ 1) to unbounded differences between B_n and D_i [–∞ ≤ (B_n – D_i) ≤ + ∞]. For the probability of a response to be used to estimate a person’ s mobility and an item’ s difficulty both sides of the equation must have the same boundaries: either 0–1 or –∞ to +∞.

The value (B_n – D_i) can be made to have the range of 0–1 by using a two-step transformation.

First, if (B_n – D_i) is expressed using natural logarithms the expression will have limits of zero and infinity:

Second, if this value [e(Bn−Di)] is expressed as an odds ratio, the expression has the limits of zero and 1 and could be a formula for the probability of a correct response:

If we take this formula as an estimate of the probability of person n responding ‘yes’ to RMI item i the relationship is:

And, therefore:

because:

Thus, for completeness:

Thus, for dichotomously scored items a general equation can be written:

where the outcome x can be either the response ‘yes’ (x = 1) or the response ‘no’ (x = 0), and e⁰ = 1. This is the Rasch model for dichotomous items.

Note. The fact that there is an x in the numerator is important when it comes to items with more than two response options (polytomous) as x can be 0, 1, 2, etc.

Andrich97 offers another way of deriving the Rasch model which exploits the relationships between odds and probabilities: the odds of a ‘yes’ response = probability of a ‘yes’ response/probability of a ‘no’ response or for short:

But, as

Thus, the odds of a ‘yes’ response =

From before:

Thus, the odds of a ‘yes’ response also=e(Bn−Di). Hence, the odds of a ‘yes’ response:

And, by taking logs:

where ln = natural logarithm. Thus, estimates of B_n and D_i are additive, and in log-odds units (logits). Taking the logarithm of a ratio scale turns it into an interval scale. Thus, the units of the person and item estimates generated by the Rasch model are interval level.

Important properties of the Rasch model

The above theory about the relationship between probability and values is logical and sensible. The mathematics seems reasonable. The important question is how this relates to a set of data, such as patients’ responses (either 0 or 1) to the items of the RMI. Before this can be explained, two fundamental properties of the Rasch model need to be demonstrated because they are required to understand the computation of item and person estimates. The first property, mentioned in the previous chapter, is that the parameters can be estimated separately. 97 The second property is that the total score is a sufficient statistic. 97 Both are explained below.

Putting it all together: relating theory and mathematics to real data

The theory that the relative locations of an item and a person govern the probability of a response is logical and sensible. The mathematics articulates this theory and has some unique properties. However, how does this relate to a set of data? How can we use the simple matrix of patients’ responses (0 or 1) to the 15 items of the RMI to generate estimates of item locations, person locations and probabilities of responses, and then go on to test the fit of the observed data to the expectations of the model?

In practice, this process is complex and requires the use of computer software. However, the general principles are relatively straightforward. Although it is best understood by considering a dichotomous scale, such as the RMI, where the item response options are either 0 or 1, the process generalises to items with more than two response options. 70,99

We know from the Rasch model that the total score for each person, i.e. the sum of one person’ s responses to the RMI items, is the sufficient statistic for computing the person location. It contains all the information required to estimate the person location. Similarly, we know that the total score for each item, the sum of the responses of all people in the sample to one item, is the sufficient statistic for computing the item location. It contains all the information required to estimate the item location.

We also know from the Rasch model that the probability of a ‘yes’ response (i.e. a score of 1) to any RMI item is given by the formula:

Likewise, the probability of a ‘no’ response (i.e. a score of 0) to any RMI item is given by the formula:

The matrix of ‘yes’ (score = 1) and ‘no’ (score = 0) responses to the RMI items allows us to compute the observed proportion of people who responded ‘yes’ and ‘no’ to each item. The observed proportion of people in a sample who respond ‘yes’ and ‘no’ to an item is an estimate of the theoretical proportion of people who respond ‘yes’ and ‘no’ to each item in the population we are studying. A theoretical proportion is a probability, because the definition of a probability of an outcome is the theoretical proportion of times that we expect that outcome to occur on a very large number of replications.

The next stage extrapolates Bernoulli’ s work (1700s) on probabilities and random variables. Essentially, if a coin is tossed multiple times the total number of heads is the sum of the theoretical average (probability) that a head will occur at each toss. Thus, conceptually, the sum of the theoretical averages (probabilities) of the number of times each item would be answered correctly should be equal to the number of items that are answered correctly. Thus, the total score for each person is the sum of the probabilities of getting each item correct and the total score for each item is the sum of the probabilities of each person getting that item correct. The probability of a person getting each item correct is given by the equation above.

Rasch analysis computer programs, such as RUMM2020, apply this logic in an iterative approach to generate estimates. Essentially, initial estimates of item and person locations are proposed. These are put into the equation above so that the probability of responding ‘yes’ can be computed for each item–person interaction (i.e. each cell in the data matrix). The probabilities are summed and this value is compared with the total score. Based on the differences between the total score and the sum of the probabilities, the initial estimates are refined. These refined estimates are put into the equation and new probabilities are computed, summed and compared with the total scores. This process continues until the sum of the probabilities is close enough to the total score. The computer program sets the criterion for ‘close’ , typically within 0.001 unit difference.

This process of iteration results in best estimates of item and person locations given the total scores for each person and item, derived from the Rasch model. The process now works backwards. From these best estimates the computer program can determine the expected value for each item–person interaction. These expected values, computed from the Rasch model, are compared with the observed scores generated by people completing items. The difference between the observed scores and the expected values is determined and indicates the extent to which the observed data fit the predictions of the Rasch model. If the data fit the model, within statistical reason, then the properties of the model hold in the data, the total scores are sufficient statistics computing item and person estimates and the estimates generated are invariant (stable) linear estimates. If the data do not fit the model, the properties of the Rasch model do not hold in the data. This provokes an investigation of items and persons to determine why.

There are many ways of evaluating the fit of the data to the model. No one method is necessary or sufficient to summarise it. More details on tests of fit are discussed in the Chapter 4.

Summary

This chapter has attempted to demonstrate the theory and mathematics of the Rasch model for dichotomous variables. The theory on which the model is grounded is logical and relatively straightforward. The mathematics articulates the theory. However, it is the properties of the mathematical model that are important. The ability to estimate the item and person locations independently of each other is a feature only of Rasch models. It is not a feature of other item response models because the additional parameters prevent it mathematically. Massof33 demonstrates this. By removing the influence of persons on item estimates, and items on person estimates, the Rasch model is able to deliver mathematically stable linear measurements of these parameters. This enables invariant comparisons of people and items to be made when the data fit the Rasch model. This is considered a requirement for fundamental measurement and is the reason why proponents of the Rasch model seek to fit data to that model, rather than try to explain observed data.

The Rasch model for items with more than two ordered response categories is a generalisation of the model for dichotomous response options. 70 The mathematics will not be discussed here. That job is best left to Andrich, who developed the model. 70,74,99 The rest of this monograph focuses on demonstrations of the application of the Rasch model to health measurement situations using the Rasch Unidimensional Measurement Models computer program. 100

Overview

The aim of this chapter is to illustrate the Rasch analysis of an existing scale, and to compare and contrast scale evaluation using traditional and Rasch psychometric methods. This chapter uses data from the CAMS study. 34 Specifically, we analyse data for the Rivermead Mobility Index (RMI). This scale has been chosen because each item has only two response categories (‘yes’ and ‘no’); that is, it consists of dichotomously scored items. It is valuable to illustrate the Rasch model in a dichotomous scale before moving on to examine its use in scales with polytomous (more than two) response options. We do this in Chapter 5.

This chapter has five sections. First, the scale is presented. Second, the sample is described. Third, we report the results of a comprehensive evaluation of the RMI using traditional psychometric statistical methods. Fourth, we report a comprehensive evaluation of the RMI using Rasch analysis. Finally, we compare and contrast the information derived from the two analyses and the inferences made about the RMI.

The Rivermead Mobility Index

Appendix 1 shows the RMI. It is a rating scale purporting to measure mobility and has 15 items. Each item concerns a mobility-related task that has two response options: ‘no’ – I am unable to do this task; ‘yes’ – I am able to do this task. The RMI is scored by clinicians from patient interview and observation. Typically, item scores are summed to give a total score that ranges from 0 (all ‘no’ responses to items) to 15 (all ‘yes’ responses to items).

Now let us consider the aims of the RMI in more detail. Mobility is being thought of (conceptualised) as a quantitative variable in the sense that it reflects a property that can have a range of values from ‘less’ to ‘more’ . The RMI items attempt to map out this idea so that responses to its 15 items can be seen as indicators of the level of mobility. Essentially, the RMI seeks to map out the mobility as a line varying from less to more on which people can be located. Thus, the 15 RMI items make operational (operationalise) the idea of the mobility variable. Because mobility is observed through a variety of manifestations, rather than directly, it is considered to be a latent (hidden) property. The words ‘trait’ and ‘construct’ are typically used instead of the word ‘property’ . Figure 1 represents graphically this conceptualisation of the mobility variable by the RMI, and the idea of measurement by locating a person on that line.

This conceptualisation implies that each item represents a mark on the ‘ruler’ of mobility mapped out by the RMI. More specifically, the mark defined by the item represents the transition point of the score from 0 to 1, i.e. the point at which a person moves from scoring ‘0’ (unable to do) to ‘1’ (able to do). The aim of a psychometric analysis is to determine, using a range of evidence, the extent to which this conceptualisation of mobility by the RMI has been achieved. In measurement terminology, the purpose of a psychometric analysis is to establish the extent to which a quantitative conceptualisation has been operationalised successfully. 3

Sample

This illustration uses data from the CAMS study. 34 The CAMS study was designed to test the notion that cannabinoids have a beneficial effect on spasticity and other symptoms related to MS. It was a multicentre, randomised, placebo-controlled clinical trial at 33 UK neurology and rehabilitation centres.

A range of outcomes were measured. The primary outcome was the Ashworth scale, a clinician-rated measure of overall spasticity. Secondary outcome measures included the RMI, Barthel Index (BI), 30-item version of the General Health Questionnaire (GHQ) and the UK Neurological Disability Scale (UKNDS).

Each patient attended eight clinic visits over 15 weeks. There were two pre-treatment pre-randomisation visits, a 5-week dose titration phase (visits 3 and 4) and an 8-week plateau phase during which people remained on a stable dose (visits 5, 6, 7). Dose reduction was carried out during week 14, and people were medication free during week 15. The final assessment (visit 8) was conducted at the end of week 15.

A total of 667 people were enrolled. Of these, 630 were treated with oral cannabis extract (Cannador, n = 211), synthetic Δ⁹-tetrahydrocannabinol (Marinol, n = 206) or placebo (n = 213). A total of 611 people were followed up for the primary end point. Treatment with cannabinoids did not have a significant effect on spasticity as assessed by the Ashworth scale.

In this illustration we used the RMI data as they have dichotomously scored items. We report the analysis of RMI data from visit 1 (including a total of n = 667 people). One person was determined to be ineligible before the RMI assessment was undertaken; hence, the data set consists of the responses for n = 666 people with MS.

Traditional psychometric evaluation of the RMI

Evaluation of the RMI using Rasch analysis

Results

Rasch analyses of the RMI data were undertaken using the software program Rasch Unidimensional Measurement Models (RUMM2020). 100 Results are reported and interpreted for each of the questions identified in the methods section, then interactively at the end.

In a Rasch analysis, people at the extremes of the scale range, those at the floor and ceiling, are excluded from the estimation of item statistics as they offer no comparison across the items to facilitate the examination of relative item difficulty. This is because people at the extremes of the scale range achieve the same score on all the items of the scale. In the CAMS sample there were n = 81 people at the extremes; thus, the item statistics were computed from n = 585.

A similar logic applies to the estimation of person locations. People at the extremes of the scale range are problematic because it is impossible to estimate their location accurately. Essentially, we do not know how far above the ceiling, or below the floor, of the scale they are. However, people at the floor and ceiling are given person location estimates in a Rasch analysis as these people are an important part of the sample and need to be included to provide a complete picture of the range of scores obtained. These estimates are achieved by extrapolation. If many extreme scores are present, this will influence the variability across the range of the test and can influence the targeting.

Is the scale-to-sample targeting adequate for making judgements about the performance of the scale and the measurement of people?

Figure 4 shows the targeting of the patient sample (top histogram) to the items (bottom histogram). The scale range has been set from −8 to +8 units (logits) for symmetry. The histogram bars represent the relative location of the item(s) and people on the same variable. At this stage we are simply trying to answer a specific question so we can consider this graph relatively superficially and return to it later.

FIGURE 4.

Targeting of the patient sample (top) to the items (bottom). Person–item threshold distribution (grouping set to interval length of 0.20 making 80 groups).

The most obvious finding is that the item locations are covered by the people, but the person locations are not covered by the items. Thus, we could infer that this is a reasonable sample to examine the scale but a suboptimal scale for measuring the sample. Hence, we would expect the scale to provide limited information about people at the extremes of the sample distribution. This is formalised by examining the information function for the scale, which is the inverse of the standard error associated with a measurement provided by the scale, at every location on the continuum: the higher the information value, the less the standard error and the greater the precision of measurement at that location. The greater the precision, the greater the information imparted by a measurement at that location. This is represented by the curve on the graph. Essentially, it informs us where on the continuum the scale performs best. This shows us that the performance of the scale is better in the centre (range −3 to +1.5 logits) and worse at the extremes. Figure 4 demonstrates that many people in this sample are located outside the best functioning of the scale.

RUMM2020 also evaluates the ‘power’ of the tests of fit as ‘excellent’ , ‘good’ , ‘reasonable’ , ‘low’ or ‘too low’ . For the RMI, this is considered excellent. This evaluation refers to the power in detecting the extent to which the data do not fit the model. 118 It does not imply that the fit of the data to the model is excellent. The power in detecting the extent to which the data do not fit the model is influenced by the targeting of the sample to the items, the PSI (explained later) and the variability of the sample. It is clearly important to interpret the fit statistics in the light of the power of the tests of fit.

The power of the tests of fit is intimately related to the PSI. If the PSI is low due to limited sample variability, i.e. people have similar locations and are not spread across the continuum, the power of the tests of fit is low. This is because lack of variability in people’ s locations makes it impossible to determine whether people with higher locations tend to get higher scores on items (the nubbins of the Rasch model). Under these circumstances, the fit statistics may appear to be very good but the power of the tests of fit will be poor.

Has a measurement ruler been constructed successfully?

Do the items map out a discernible line of increasing intensity?

Table 7 shows the 15 RMI items ordered by their location (also called calibration), in ascending order (from the most negative to the most positive). They range from about −3 to +6 logits, which is a wide range. Thus, the items define a line of increasing intensity, a continuum, rather than just a point.

TABLE 7 - Item locations in ascending order (n = 585; 666 with 81 extremes excluded)

RMI, Rivermead Mobility Index; SE = standard error.

RMI item	Item location	Location SE
1. Turning over in bed	–3.032	0.130
3. Sitting balance	–2.781	0.126
2. Lying to sitting	–2.707	0.125
6. Transfer	–2.423	0.122
4. Sitting to standing	–1.863	0.117
8. Walking inside, with aid if needed	–1.568	0.116
13. Bathing	–0.872	0.115
5. Standing supported	–0.619	0.115
9. Walking outside (even ground)	0.444	0.119
7. Stairs	0.515	0.119
11. Picking off the floor	0.940	0.122
10. Walking inside, no aid	2.566	0.144
12. Walking outside (uneven ground)	2.641	0.146
14. Up and down four steps	2.715	0.148
15. Running	6.042	0.356

Items have both negative and positive values because the mean item location is always set to 0 to give an arbitrary origin. It is used as a constraint because Rasch analysis estimates the locations of the items and people relative to each other, and not their absolute locations. That is, in the case of any two items, only the difference in their locations, not an independent value for each item, can be estimated. Thus, to give the items and people absolute values a constraint is required – the convention is that the mean of the item locations is set to 0. This explains why Rasch (and IRT) analyses typically have the curious finding of values ranging from negative to positive.

Table 7 also shows the standard error associated with each item location estimate. These vary; specifically, standard errors are smallest at the centre and largest at the extremes. They are also influenced by the size of the study sample, and the targeting of the scale to the sample. This is because the formula for the standard error (SE) is given as [SE = 1/√ sum of p(1 – p) for each person in the sample]. At the heart of this equation is 1/p (1 – p). This is smallest when p = 0.5, i.e. when the proportion of people responding ‘yes’  = proportion of people answering ‘no’ = 50%. As p moves away from 0.5, either down towards 0 or up towards unity, the value of 1/p(1 – p) gets larger.

Figure 5 complements Table 7 by showing the item locations graphically. It shows that the 15 RMI items are not evenly spread. They are bunched in two ways. First, items are bunched such that at three places on the continuum multiple items have the same location. These are marked with thin arrows. Table 7 confirms the items with the very similar locations: ‘sitting balance’ (−2.781) and ‘lying to sitting’ (−2.707); ‘walking outside on even ground’ (+0.444) and ‘stairs’ (+0.515); and ‘walking outside on uneven ground’ (+2.641) and ‘up and down four steps’ (+2.715). Items with similar locations raise the possibility of one of the items being redundant.

FIGURE 5.

Targeting of the patient sample providing a representation of the measurement ruler mapped out by the 15 RMI items. Person–item threshold distribution (grouping set to interval length of 0.20, making 80 groups).

Items are also bunched such that there are notable gaps in the continuum mapped out by the items between +3 and +6 units, +1 and +2.5 units, and −0.5 and +0.5 units. These are marked with thick arrows. Gaps imply limited measurement at those areas on the continuum they attempt to map out, despite the fact that they spread over a reasonable range (−3 to +6 units).

Figure 5 provides a clear representation of the measurement ruler mapped out by the 15 RMI items. Its adequacy and limitations are explicit. This figure acts as an evidence base for improvement of the RMI. For example, it would benefit from including items that are more difficult than going up and down four steps (location +2.715) but not as difficult as running 10 metres (location +6.042). It is clear to see how such a figure could be invaluable during scale construction.

Is the location of items along this line reasonable?

The continuum mapped out by the items warrants some explanation. On this continuum, items to the left indicate those items that are ‘easy’ mobility tasks to do whilst items to the right indicate those that are ‘hard’ mobility tasks to do. This is because high scores on the RMI indicate people who answer more ‘yes’ more times (i.e. can do more tasks) and thus have relatively lower disability. In contrast, low RMI scores (more ‘no’ responses) indicate higher disability. Thus, as people become progressively more disabled they move along the disability continuum from right to left, and if people improve they move from left to right.

In contrast, a high score for an item indicates more people who answer ‘yes’ (can do), and thus means the item is relatively easy. A low score for an item indicates more people who answer ‘no’ and a difficult item. Thus, the continuum for items runs from the easiest task to do on the left (‘turning over in bed’) to the most difficult task to do on the right (‘running’). This makes sense: as people become progressively more disabled their RMI score falls and they move from right to left on this continuum, and as they become disabled the first task they find difficult is the most difficult (‘running’). If the scoring had been the other way round (0 = can do; 1 = can’ t do), or for scales on which high scores indicate greater disability (e.g. MSIS-29), the direction of the continuum would be reversed.

The ordering of the 15 RMI items is clinically reasonable. Of the 15 items, ‘running’ (location +6.042) is predicted to be the most difficult and ‘turning over in bed’ (location −3.032) the easiest. Also, for example, the relative ordering of items as ‘walking inside with an aid’ (location −1.568) before ‘walking inside without an aid’ (location +2.566), and ‘walking outside (even ground)’ (location +0.444) before ‘walking outside (uneven ground)’ (location +2.641), makes clinical sense.

Do the items work together to define a single variable?

A central theme of a Rasch analysis is an examination of the ‘fit’ of the observed data to the expectations of the mathematical model. By fit we mean the extent to which the observed responses of persons to items are predicted by, or recovered from, the mathematical model.

To recap, the Rasch model is a mathematical expression of the requirements that rating scale data must meet to generate values that can be considered measurements. This means that the Rasch model defines the relationships between item locations and person locations that should exist if a set of items is to deliver on the key tenets of measurement: invariance, unidimensionality and interval-level estimates.

What happens in practice? The Rasch model uses the total score for people (achieved by summing the scores of the items each person responds to) and the total scores for items (achieved by summing the scores of the persons who responded to each item) to derive best estimates of person mobility and item difficulty. As we have seen, this is legitimate, mathematically, because the total score can be proven to be the sufficient statistic for estimating person mobility and item difficulty. Having derived these estimates, the analysis now works backwards. It uses these estimates to derive the predicted responses that should have occurred for the items and persons to satisfy the Rasch model. All that remains is to compare the observed responses of the patients to the RMI items with the predicted responses derived from the model. Fit indicators essentially summarise the extent of the difference between the observed and expected responses.

In the Rasch paradigm, there is no one indicator of fit of the observed data to the mathematical model that is necessary and sufficient to summarise fit. There are multiple methods. Each addresses a different aspect of fit, or fit from a different perspective. For this reason, all available fit indicators should be examined; but their interpretation should be simultaneous and interactive rather than singular and in isolation of each other. However, in presenting the data for the RMI below it is necessary to consider each indicator separately before interpreting them interactively. For dichotomously scored items RUMM2020 provides two numerical (fit residual and chi-squared) and one graphical (item characteristic curve, ICC) indicators of fit.

Fit residual

The fit residual (sometimes called the log residual) evaluates the fit of the observed data to the Rasch model from the perspective of the items. 118 For each item, the fit residual summarises the interaction between that item and all the persons for whom there is a response to that item. More specifically, the fit residual for an item is a summary of the differences between observed and expected values from each and every response to it (item–person interaction).

For every item–person interaction there is an observed score (0 or 1 for the RMI) and an expected score derived from the Rasch model (any value between 0 and 1). The difference between the observed and expected scores is called a residual (observed – expected = residual). For each item of the RMI, the residuals from the interactions with each of the n = 585 people in the sample are squared, summed and transformed to give a summary value (the fit residual) with possible range of –∞ to +∞.

Fit residuals are standardised to approximate a standard normal deviate. This means that the fit residuals are expected to be normally distributed with mean = 0 and SD = 1. This is based on the hypothesis that if the data fit the model the deviations (residuals) between the observed responses and the model-derived expected values should be no more than random errors. Therefore, if the data fit the Rasch model the mean fit residual, across all the items should be close to 0, the SD close to 1, and the individual values for the items should be distributed in the approximate range −2.5 to +2.5 (more specifically, 99.5% of values in the range −2.5 to +2.5, and 99.9% in the range −3.0 to +3.0).

For individual items, an observed fit residual of 0 indicates no difference between observed and predicted scores (perfect fit). The greater the departure from 0 (regardless of accompanying + or – sign), the greater the discrepancy between observed and predicted responses, and thus the greater the misfit of the observed data to the model. The sign (+ or –) associated with the fit residual value alludes to the type of misfit. Negative values indicate overdiscriminating items relative to the model, while positive values indicate underdiscriminating values relative to the model. This will be explained in more detail later.

Table 8 shows the fit residuals for the 15 RMI items in ascending order. The mean item fit residual is −0.8, the SD 1.7 and the range −3.775 to +2.188. This indicates there is some misfit of the observed data to the Rasch model that needs to be explored and explained. Four items (11, 5, 9 and 8) have fit residuals outside the range of −2.5 to +2.5, indicating that the observed responses to these four items are not consistent with those predicted by the Rasch model. It is noteworthy that fit residuals for three of these four items (5, 9 and 8) lie within the range −3.0 to +3.0 and thus only a small amount outside the recommended range. Hence, only item 11 (‘picking off the floor’) fails notably this one criterion of fit. The implications of this will be explored later.

TABLE 8 - Items in fit residual order (n = 585; 666 with 81 extremes excluded)

RMI, Rivermead Mobility Index; SE, standard error.

RMI item	Item location	Location SE	Fit residual
11. Picking off the floor	0.940	0.122	–3.775
5. Standing supported	–0.619	0.115	–2.850
9. Walking outside (even ground)	0.444	0.119	–2.750
8. Walking inside, with aid if needed	–1.568	0.116	–2.729
6. Transfer	–2.423	0.122	–1.514
7. Stairs	0.515	0.119	–1.143
2. Lying to sitting	–2.707	0.125	–0.803
14. Up and down four steps	2.715	0.148	–0.759
12. Walking outside (uneven ground)	2.641	0.146	–0.503
15. Running	6.042	0.356	–0.202
1. Turning over in bed	–3.032	0.130	–0.031
10. Walking inside, no aid	2.566	0.144	0.088
13. Bathing	–0.872	0.115	1.217
3. Sitting balance	–2.781	0.126	1.626
4. Sitting to standing	–1.863	0.117	2.188

Chi-squared value and its probability

The chi-squared value is an indicator of the interaction between the individual item and the trait measured by the set of items118 (here mobility measured by the 15 RMI items). It is much easier to understand the chi-squared test of fit by seeing both the numerical value and a visual representation. This will be done by referring to Table 9a, which shows the 15 RMI items ordered by increasing chi-squared value, and Figure 6, which is the item characteristic curve (ICC) (explained below) for item 1, whose chi-squared value in Table 9a is 4.557.

FIGURE 6.

Item characteristic curve for item 1 including class intervals.

TABLE 9a - Rivermead Mobility Index (RMI) items in chi-squared probability order (n = 585; 666 with 81 extremes excluded; six class intervals)

SE = standard error.

RMI item	Item location	Location SE	Fit residual	χ2 value	χ2 probability
14. Up and down four steps	2.715	0.148	–0.759	3.971	0.5536
1. Turning over in bed	–3.032	0.130	–0.031	4.557	0.4722
12. Walking outside (uneven ground)	2.641	0.146	–0.503	4.760	0.4459
10. Walking inside, no aid	2.566	0.144	0.088	7.322	0.1978
13. Bathing	–0.872	0.115	1.217	8.770	0.1186
15. Running	6.042	0.356	–0.202	10.973	0.0519
9. Walking outside (even ground)	0.444	0.119	–2.750	11.453	0.0431
7. Stairs	0.515	0.119	–1.143	11.585	0.0409
5. Standing supported	–0.619	0.115	–2.850	13.742	0.0173
4. Sitting to standing	–1.863	0.117	2.188	17.230	0.0041
11. Picking off the floor	0.940	0.122	–3.775	25.877	0.0001
8. Walking inside, with aid if needed	–1.568	0.116	–2.729	28.581	0.0000
6. Transfer	–2.423	0.122	–1.514	29.498	0.0000
2. Lying to sitting	–2.707	0.125	–0.803	31.761	0.0000
3. Sitting balance	–2.781	0.126	1.626	51.566	0.0000

The chi-squared value in Table 9a is a summary statistic. It is computed by summing the chi-squared values for a series of class intervals. A series of class intervals is achieved by dividing the sample into a number of similarly sized groups based on their level of disability. For each class interval, the mean location of the people and their mean score on each of the 15 items are computed. Then, for each of the 15 items, the mean observed scores for the class intervals are compared with the scores for those items predicted by the Rasch model at the mean location of the class interval. The chi-squared value for each item is the sum of the chi-squared values computed for each of the six class intervals. The associated chi-squared probability is the probability that the discrepancy between the observed mean and the expected value is large relative to chance. If the chi-squared value is significant (p < 0.05 or 0.01) the item should be examined.

Figure 6 shows this process graphically. The S-shaped curve is the ICC for item 1. This indicates the expected score (y-axis) on item 1 for each possible location on the mobility continuum (x-axis). Note that the expected score can be any value between 0 and 1, unlike the observed responses, which can only be 0 or 1. The six small vertical marks on the x-axis indicate the mean person locations for each of the six class intervals. The six black dots indicate the observed mean score on item 1 for each of these six class intervals. The chi-squared values in Table 9a (4.577 for item 1) summarise the coherence between the observed responses (black dots) and expected responses (ICC) at the six points on the continuum. They are computed by summing the component chi-squared values for each class interval (Table 9b).

TABLE 9b - Item 1: component chi-squared values for six class intervals

ES, expected score; EV, expected value; OM observed mean.

Item: χ² = 4.557 (df = 9; p = 0.472249).

Class interval		Location		Component			Category responses
No.	Size	Max	Mean	Residual	χ2		0	1
1	97	–3.170	–3.544	1.555	2.148	Obs. p	0.55	0.45
						Est. p	0.63	0.37
[OM = 0.45; EV = 0.38; OM–EV = 0.08; ES = 0.16]						Obs. t		0.45
2	79	–2.097	–2.310	0.486	0.237	Obs. p	0.30	0.70
						Est. p	0.33	0.67
[OM = 0.70; EV = 0.67; OM–EV = 0.03; ES = 0.05]						Obs. t		0.70
3	90	–1.105	–1.387	0.804	0.646	Obs. p	0.13	0.87
						Est. p	0.16	0.84
[OM = 0.87; EV = 0.84; OM–EV = 0.03; ES = 0.08]						Obs. t		0.87
4	86	–0.053	–0.334	–0.620	0.385	Obs. p	0.08	0.92
						Est. p	0.06	0.94
[OM = 0.92; EV = 0.94; OM–EV = –0.02; ES = –0.07]						Obs. t		0.92
5	100	1.119	0.846	–0.615	0.378	Obs. p	0.03	0.97
						Est. p	0.02	0.98
[OM = 0.97; EV = 0.98; OM–EV = –0.01; ES = –0.06]						Obs. t		0.97
6	133	4.716	2.896	0.703	0.494	Obs. p	0.00	1.00
						Est. p	0.00	1.00
[OM = 1.00; EV = 1.00; OM–EV = 0.00; ES = 0.06]						Obs. t		1.00

It is important to note that the number of class intervals can be altered. The RUMM program will select an appropriate number of class intervals (possible range 2–10) based on the size of the study sample. The analyst can override this default. Altering the number of class intervals will alter the chi-squared values, but is less likely to alter the associated probability and inferences. We find it informative to examine the implications of changing the number of class intervals.

A number of facts need to be considered when interpreting the chi-squared values. Chi-squared values increase with sample size. Hence, the developers of RUMM2020 recommend that when using large samples the sample size is amended to n = 500 to compute the values. 118 Chi-squared values are affected by the number of class intervals chosen, as discussed above. Chi-squared values only approximate a chi-squared statistic, and are inflated when the estimated probabilities are close to 0 or 1.

For these reasons, Andrich et al. 118 suggest that it is best to use the chi-squared statistic as an order statistic (i.e. order of degree of misfit) to see which items show much greater values than others, and to examine the ICC (see below). Consider Table 9a in which the items are ordered by ascending chi-squared values. They range from 3.971 (item 14, ‘up and down four steps’) to 51.566 (item 3, ‘sitting balance’). Andrich et al. recommend examination of the values and how they change sequentially across items. For the RMI, there is a gradual increase over the first 10 items from item 14 (χ² = 3.971) up to item 4 (χ² = 17.230), then there is a larger step (item 11 = 25.877), followed by a fairly gradual increase for the next three items (χ² = 28.581–31.761) before a large step to the final item (item 3, χ² = 51.566). The chi-squared probabilities are significant at the 0.01 level for the last six items (4, 11, 8, 6, 2, 3) in Table 9a. Of these items, item 3 is notably the most wayward.

Item characteristic curve

The third indicator of observed data-to-Rasch model fit generated by RUMM2020 is the ICC. This is a graphical rather than a numerical indicator of fit and it aids the interpretation of the two fit statistics detailed above.

The ICC is a graph for an individual item. It plots the expected response (predicted from the model) to an item at each and every level of the measurement continuum. Clearly, the only available responses are 0 (no) and 1 (yes). However, the potential responses are continuous and vary from 0 to 1.

Figure 7 shows the ICC for item 1 (‘turning over in bed’). The ICC plots the expected values for the item (predicted by the Rasch model) on the y-axis against the location on the RMI-measured mobility continuum. Essentially, as a person moves from left to right on the x-axis, i.e. from more disabled (low RMI score) to less disabled (high RMI score), his or her expected value on item 1 (in fact any of the items) increases.

FIGURE 7.

Item characteristic curve for item 1 excluding class intervals.

When the class intervals are added to the plot they appear as black dots (see Figure 6). In this example we have chosen six class intervals. Each dot represents the intersection between the item mean score for the people in the class interval (y-axis) and the mean person location on the mobility continuum for the class interval (x-axis). The closer the dots follow the ICC, the better the fit of the observed data to the predictions of the Rasch model. As we have discussed, the chi-squared statistics in Tables 9a and 9b give the numerical values.

For item 1 the dots representing the six class intervals follow the ICC very well. Hence the chi-squared values are small. However, it is noteworthy that in Figure 6 there is only one class interval whose mean is in the lower half of the continuum covered by the item. This situation is not altered substantially by increasing the numbers of class intervals. Thus, this sample is not a stringent test of item 1’ s performance across the range of the continuum. The best test of an item is a sample that has class intervals across the item’ s range (e.g. item 5; Figure 8).

FIGURE 8.

Item characteristic curve for item 5 including class intervals.

The importance of the graphical indicator of fit is that it gives perspective to the numerical values in Tables 8 and 9a. Chi-squared values and their associated probabilities can appear daunting and worrying when considered in isolation of the graph, which visualises the extent to which each class interval departs from expectation. However, this plot is rich with other information. It shows the suitability of the sample for examining the items. For example, the statistics may be excellent because the sample is poorly targeted to the item (Figure 9; the ICC for item 15).

FIGURE 9.

Item characteristic curve for item 15 including class intervals.

The plot of the ICC and class intervals also helps to interpret the + or – sign associated with fit residual values. Positive fit residuals occur when the slope of the line from the class intervals is flatter than the predicted ICC – this is interpreted as underdiscriminating (Figure 10; ICC with class interval for item 3). When the slope of the line from the class intervals is steeper than the predicted ICC, this means that the item is overdiscriminating (Figure 11; ICC with class interval for item 7).

FIGURE 10.

Item characteristic curve for item 3 including class intervals.

FIGURE 11.

Item characteristic curve for item 7 including class intervals.

General comments about fit indicators

In addition to the specific comments associated with each fit indicator the interpretation of fit statistics needs to consider a number of facts:

1. The predictions of a mathematical model are predictions of perfection. Observed data are associated with limitations of various origins. Therefore, misfit is to be expected. The key issue is what does it mean and does it matter? Thus, the degree of fit must be interpreted in the light of the goals of measurement and evaluation. For example, many established scales were developed using either traditional psychometric or no psychometric methods. Rasch analysis is more likely to identify problems with those scales than with scales constructed using Rasch methods (when applied appropriately). The immediate response might be to change existing scales – but that might not be the best way forward.

2. No one fit indicator is necessary and sufficient to summarise fit. Andrich recommends that the results of fit indicators are interpreted interactively and together, rather than sequentially and in isolation (much like the results of a clinical assessment). 119

3. Fit statistics inform us that the responses to items are not as predicted. They do not diagnose the cause of the misfit. It is the analyst’ s job to try and explain why misfit occurs. The knee jerk response of removing items that misfit is to be avoided.

4. Fit statistics, like all Rasch statistics, give an indication of fit within the frame of reference of the item set. Thus, removing and adding items may change the values achieved.

5. Changing the number of class intervals will change the chi-squared values and the black dots on the ICC (but not the ICC itself). It is likely to have less effect on the associated probability and inferences made. Changing the number of class intervals will not affect the fit residual value.

6. Items that anchor the extremes of the scale range might have somewhat erroneous fit statistics.

7. Poor targeting may under- or overestimate fit statistics.

Interactive interpretation of fit indicators for the RMI

What does this all mean for the 15 RMI items? Table 10 summarises the results of the three indicators of fit in a very literal sense (pass/minor fail/major fail). This table shows that six items (1, 10 and 12–15) pass all three fit criteria, two items (8 and 11) fail all three criteria, two items (2 and 3) fail two criteria and the remaining five items (4–7 and 9) fail one criterion. Only three of the 15 criterion failures are really notable: item 11’ s fit residual at −3.775 and item 3’ s chi-squared value (51.566) and adherence to the ICC. The remaining departures from expectation are relatively small.

TABLE 10 - Summary of item fit statistics

F+, minor fail; F++, major fail; ICC, item characteristic curve; P, pass.

Item	Fit residual	χ2	ICC	F+	F++
1	P	P	P	0	0
2	P	F+	F+	2	0
3	P	F++	F++	0	2
4	P	F+	P	1	0
5	F+	P	P	1	0
6	P	P	F+	1	0
7	P	P	F+	1	0
8	F+	F+	F+	3	0
9	F+	P	P	1	0
10	P	P	P	0	0
11	F++	F+	F+	2	1
12	P	P	P	0	0
13	P	P	P	0	0
14	P	P	P	0	0
15	P	P	P	0	0

The implication of misfit is that it undermines the inferences made from the data. Essentially, for the total score to be a sufficient statistic, and for the estimates of items and persons generated to be invariant, and on an equal interval scale, the data need to fit the Rasch model. The question then becomes to what extent does the misfit associated with a specific analysis disrupt this process?

One common approach to the problem of misfitting items is to remove them in order to create a modified RMI with better fit statistics. A better approach is to try to diagnose why misfit has occurred in some items given that the 15 items of the RMI are conceptually related to mobility – the construct the RMI seeks to measure.

Item 11 (‘picking off the floor’) failed all three item fit criteria. The fit residual is negative, indicating an overdiscriminating item relative to the model. The ICC (Figure 12) shows this – the slope of the item (rate of change of item score across the continuum) is steeper than the model requirement. The class interval dots tend to be below the line at the left-hand end and above the line at the right-hand end. Thus, among people with greater levels of disability, more people than predicted were unable to do this task, and, among people with lesser levels of disability, more people than predicted were able to do the task. One explanation of the misfit is that this item involves more than mobility, and is less related to mobility than the other items. Another explanation is that there may been some ambiguity, or that the items may not be a common task for people to do. So, there may be some estimating by people as to their ability to do the task – or the interviewers may set different standards for success and failure.

FIGURE 12.

Item characteristic curve for item 11 including class intervals.

Item 7 (‘stairs’) also failed all three item fit criteria and demonstrated a similar pattern of fit results to item 11. As it stands, item 7 is ambiguous in that the number of stairs constituting a flight varies widely. Some people might not climb their flight of stairs as there are too many steps. So, they would report unable when they are able to do a few steps – which for others might constitute a flight. In addition, there is no clarification of ‘help’ , which can be in many forms (handrail, verbal encouragement, hands-on help from another person). Thus, there are many explanations for misfit, and were one to modify the RMI it would be appropriate to address these issues before removing items. It is perhaps surprising to note that managing a flight of stairs (location = −1.568) appears to be considerably easier than going up and down four steps (location = +2.715).

Items 3, 7 and 11 came out worst. All three items have some degree of ambiguity. Item 3 involves a judgement of time and is not a regular task. It is uncertain why 10 seconds was used as the criterion. Item 7 does not define what constitutes a ‘flight of stairs’ or ‘help’ . However, it is interesting that item 3 (‘sitting balance’) failed two item fit criteria. It had the largest chi-squared values and therefore chi-squared probability. The ICC (see Figure 10) shows that the slope of the observed responses is flatter than the ICC. Thus, the item is less discriminating than the model requirements. This explains the positive fit residual. Thus, at the more disabled end of the continuum, more people are able to do this task than predicted. Also, people in four successive class intervals, and by definition with different levels of disability, tend to get the same mean score on the item. One explanation is that item 3 may be difficult to interpret.

Does the response to one item directly influence the response to another?

The Rasch model requires that the items are locally independent, i.e. the response to one item in a scale is independent of the response to another in the scale. In a Rasch analysis, one way of studying local independence stems from the theory that the residuals (observed –  expected values) should represent random error. Therefore, the residuals for the items should not be correlated. If they are correlated the implication is that the answer to one item is dependent on another. Thus, the correlations among the residuals should be low. The criterion of < 0.30118 has probably been chosen as this represents 10% shared variance. For the RMI, in this sample, none of the residual correlations exceeded 0.30. This implies that the responses of the items are independent of each other and that the items are locally independent.

Andrich has recently developed another way of examining for local independence. This method is not yet freely available or published. Essentially, this method involves combining items that might be dependent to form a single item with more response categories (he has named these subtests), and re-examining the modified scale. If the PSI of the modified scale falls notably relative to the original scale this implies that the reliability of the original scale was inflated by the item dependency. The fact that item dependency inflates reliability estimates has long been known. The importance of this method lies in the fact that the residual correlations do not always identify item dependency. It is therefore superior both conceptually and empirically.

We examined for local dependency using this new method. Specifically, we examined the implications of combining two pairs of items that could be dependent: ‘walking outside (even ground)’ with ‘walking outside (uneven ground)’ ; and ‘walking inside with an aid if needed’ with ‘walking inside with no aid’ . Each pair of items was combined to form one item with four response options using the subtest facility within RUMM2020. The PSI was unaffected, implying no significant local dependency.

Are the locations of the items stable across clinically important groups?

In this example we examined item functioning across the three treatment arms of the study (placebo, Cannador, Marinol). The three groups are clinically important because they represent three different subsamples of the study. Whether these randomly generated samples are statistically equivalent in terms of their clinical characteristics is an empirical question. Whether these samples ‘handle’ the scale in the same way is a further empirical question. There was no evidence that the three groups handled any of the items differently. Thus, we have evidence that item performance across the three treatment groups is stable and that the three groups can be measured on a common ruler. There is a discussion of DIF and how to examine and interpret it in Chapter 6.

Have the people in the sample been measured successfully?

Are the persons in the sample separated along the line defined by the items?

Figure 4 shows the distribution of person measurements (locations) relative to the item locations. The sample is well spread with values ranging from around −4.75 to +7 logits. The mean is −0.861 (SD 2.622), indicating that the sample is off-centre of the items (as the mean of the item locations is always 0).

Figure 4 is a graphical indicator that the items of the scale have been successful in separating this sample of people with MS. One numerical indicator of the degree of separation is the PSI. 120 This is computed from the person location estimates as the variation among person locations relative to the error of estimate for each person. Thus, it is consistent with the traditional definition of reliability of a scale, i.e. how reliably the scale distinguishes between the responders. The PSI tells us how much of the variation of person estimates can be attributed to error variances, i.e. the extent to which scores are associated with random error. Thus, the PSI is a reliability indicator, and like most reliability indicators ranges from 0 (all error) to 1 (no error). The fact that it focuses (in both name and computation) on the separation of the persons indicates that the PSI is not a property of the scale but a property of the scale in relation to the specific sample of persons measured. In contrast, the analogous reliability statistic of traditional psychometric analysis, Cronbach’ s alpha, has the same formulaic structure as the PSI, but relates the variance among persons to the variance of the items.

It is interesting to note that the PSI and Cronbach’ s alpha often produce near identical values. Indeed, RUMM2020 reports Cronbach’ s alpha (partly to satisfy those people who appeared to want the reassurance of at least seeing this index, perhaps as it is the main reliability estimate for traditional psychometric analyses121). However, there are fundamental differences between the PSI and coefficient alpha. First, the PSI is expressed entirely in terms of person locations and so meets the true definition of reliability. Second, the PSI is computed from linear measurements rather than raw scores. Third, the PSI can be computed when data are incomplete (i.e. there are missing item responses) whereas alpha requires complete data (i.e. it is computable only on the subsample with complete data). This is because, in Rasch analysis, missing responses affect the standard error of a person location not the ability to generate an estimate. The PSI will, however, decrease. Further details about the PSI can be found in Andrich’ s paper. 120

In the RMI data set, both PSI and alpha are reported as there are no missing data. The values are essentially identical at 0.91 indicating good reliability. It is important to note that the PSI, like alpha, is sample dependent because it is computed from the person locations. Essentially, it indicates the ability of a set of items to separate the study sample. Thus, the PSI is a function of the data, not an independent function of the scale.

There are some notable features about the distribution of the people in the sample. The sample is not normally distributed. This is neither expected nor wanted, as the distribution of the sample is an empirical finding rather than a requirement. However, it does have implications by suggesting it would be advantageous not to make assumptions about the distribution of samples and traits in populations. The largest frequency of patients (n ≈ 80, ≈ 12%) is at the floor of the scale range, and about 50% are within the lower third of the scale range. This provides further evidence of suboptimal targeting of the RMI to the study sample, especially in the context of a clinical trial, when the ability to detect change is paramount.

Figure 13 shows the plot of RMI total scores against the intervalised measurements they imply. The curve is S-shaped, although not substantially so. Table 11 shows the measurement implied by each RMI total score. These are the best estimates and can be used when people have complete data. Also tabulated are the changes in interval measurement units implied by each single-point change score. These vary 4.7-fold across the range of the scale.

FIGURE 13.

Plot of RMI total scores against the intervalised measurements they imply.

TABLE 11 - Rivermead Mobility index (RMI) raw scores and the interval measures they imply

NA, not applicable.

This column represents the change in interval-level measurement implied by a 1-point difference in RMI raw score. For example, a change in RMI raw score from 0 to 1 implies a change in interval-level measure of 0.898 logits (*).

RMI raw score	Interval measure	Standard error	Change implied by 1-point differencea
0	–4.780	1.358	NA
1	–3.882	0.984	0.898*
2	–3.170	0.824	0.712
3	–2.607	0.755	0.563
4	–2.097	0.725	0.510
5	–1.603	0.715	0.494
6	–1.105	0.718	0.498
7	–0.590	0.729	0.515
8	–0.053	0.746	0.537
9	0.513	0.768	0.566
10	1.119	0.798	0.606
11	1.779	0.837	0.660
12	2.496	0.896	0.717
13	3.323	1.018	0.827
14	4.713	1.374	1.390
15	7.047	2.406	2.334

Table 11 also gives the standard error associated with each of the 16 possible RMI person locations. This is a function of the number of items, the relative relationship of the item locations to the person locations (targeting) and the person’ s total score. The greater the number of items a person responds to, and the better the targeting, the smaller the standard error. The reason, as for the standard error of item estimates, is that the formula for computing the standard error of a person’ s location estimate is {SE = 1/√[sum of p(1 – p) across items answered]}.

Figure 14 plots the SE against each RMI location when there are complete data (i.e. all items have been answered). The curve is U-shaped and indicates that standard errors vary threefold across the range of the scale, and are greatest at the extremes and smallest at the centre of the scale range. This is logical. People who score at the floor and the ceiling are those for whom we have the least confidence about their estimate – we do not know how far above or below the ceiling or floor they really lie.

FIGURE 14.

Plot of standard error against location.

There are two implications of the U-shaped curve in Figure 14. The first is that the most precise measurement occurs in the centre of the range covered by a scale. The second implication concerns the measurement of change. The statistical significance of change is influenced by a person’ s location at each measurement time point (e.g. pre and post treatment) as well as the size of the change score.

It is noteworthy that in Table 11 each RMI raw score implies one linear measurement. Some are concerned by this as it implies that no regard is given to the combination of items that are ‘passed’ or ‘failed’ . Consider an RMI raw score of 7. In practice, this can be achieved by scoring 1 for any seven RMI items. The extreme situation would be two people who attain an RMI total score of 7: person A scores 1 for each of the seven easiest items (i.e. items 1, 3, 2, 6, 4, 7, 8 and 13), while person B scores 1 for each of the seven hardest items (i.e. items 9, 7, 11, 10, 12, 14 and 15). These two people achieve the same RMI total score, the same RMI linear measure and the same standard error. This concerns some people, who argue that a person’ s measurement should be influenced by which specific items they ‘passed’ and not simply by their aggregate score. They would expect person B to have a higher location (i.e. better mobility) than person A, and so might choose another item response model to analyse their rating scale data. In the Rasch model, the reason that persons A and B get the same location results from one of the mathematical properties of the model: that the total score is the sufficient statistic for estimating a person’ s location. This was explained in Chapter 3. This means that there is no more information in the response pattern for determining their location. This does not mean that the pattern of responses isn’ t important. On the contrary, it is fundamentally important and accounted for in the person fit statistic – the extent to which a particular individual’ s responses to the 15 RMI items accord with the expectations of the measurement model (explained below). Person B would be identified as a marked ‘misfit’ that would require explanation before his or her measurement was considered for further analysis. From a Rasch perspective, the pattern of responses invokes enquiry as to why such an unlikely pattern of responses occurs, rather than invoking a change in the estimate of that person’ s location.

Also, note that Table 11 is for complete data only. That is, the person location implied by each total score (e.g. a total score of 7 implies a location of −0.590) is applicable only for a person who has responded to all 15 items. It is important to be certain about what happens when there are missing item responses (i.e. incomplete data). Clearly, tables cannot be produced for all eventualities. For example, consider person C, who achieved a total score of 7 but responded only to 10 items; the other items where left blank. This person’ s location would differ from that of a person with complete data and a total score of 7, and the standard error would differ (as this is related to the number of items). The key issue here is that the Rasch analysis uses the available data to generate an estimate and does not make assumptions about the missing data.

Do individual placements of people on the variable make sense?

This analysis did not examine correlations with other measures, group differences, hypothesis testing and clinical ordering of persons in terms of their relative locations.

How valid is each person’ s measurement?

The item fit residual, discussed above, summarises the extent to which responses to each individual item are consistent with those expected by the Rasch model. This value is achieved for each of the 15 RMI items by summarising the residuals arising from 585 patients’ responses to that item. Similarly, we can achieve a value for each of the 667 patients by summarising the residuals from each person’ s responses to the 15 RMI items. This is called the person fit residual; it summarises the extent to which responses by each person are consistent with those expected by the Rasch model, and is used to identify misfitting individuals.

As before, the residual is produced by subtracting the expected score from the observed score. The predicted scores are calculated from the Rasch model using the estimates of the person and item locations. The residual is standardised by dividing it by the square root of the variance, which is computed from the expected value (EV) using the following formula: variance = EV – EV2. This process leads to a standardised fit residual for each person’ s response to each item and these are transformed and summarised to form the person fit residual which approximates a standard normal deviate. Thus, for each person the fit residual should ideally lie within the range −2.5 to +2.5.

In this sample, person fit residuals ranged from −1.042 to +1.969 (mean −0.224; SD 0.421). Thus, none outside the range −2.5 to +2.5. Tables 12 and 13 show the observed and expected values for one of the better and one of the worse fitting persons. Note that the fit residual for the better fitting person (Table 12) is −1.042, despite a perfect response ‘pattern’ . This is because the model expects some departures from perfection – which is seen as overfitting.

TABLE 12 - Observed and expected responses for one of the better fitting persons (n = 585; 666 with 81 extremes excluded)

RMI, Rivermead Mobility Index; SE, standard error.

Person location  = 1.779; SE = 0.837; fit residual = –1.042.

RMI item	Item location	Location SE	Observed score	Expected value	Fit residual
1. Turning over in bed	–3.032	0.130	1	0.992	0.090
3. Sitting balance	–2.781	0.126	1	0.990	0.102
2. Lying to sitting	–2.707	0.125	1	0.989	0.106
6. Transfer	–2.423	0.122	1	0.985	0.122
4. Sitting to standing	–1.863	0.117	1	0.974	0.162
8. Walking inside, with aid if needed	–1.568	0.116	1	0.966	0.188
13. Bathing	–0.872	0.115	1	0.934	0.266
5. Standing supported	–0.619	0.115	1	0.917	0.301
9. Walking outside (even ground)	0.444	0.119	1	0.792	0.513
7. Stairs	0.515	0.119	1	0.780	0.532
11. Picking off the floor	0.940	0.122	1	0.698	0.657
10. Walking inside, no aid	2.566	0.144	0	0.313	–0.675
12. Walking outside (uneven ground)	2.641	0.146	0	0.297	–0.650
14. Up and down four steps	2.715	0.148	0	0.282	–0.626
15. Running	6.042	0.356	0	0.014	–0.119

TABLE 13 - Observed and expected responses for one of the worse fitting people (n = 585; 666 with 81 extremes excluded)

RMI, Rivermead Mobility Index; SE, standard error.

Person location  = 1.119; SE = 0.798; fit residual = 1.969.

RMI item	Item location	Location SE	Observed score	Expected value	Fit residual
1. Turning over in bed	–3.032	0.130	0	0.984	–7.966
3. Sitting balance	ndash;2.781	0.126	1	0.980	0.142
2. Lying to sitting	–2.707	0.125	1	0.979	0.148
6. Transfer	–2.423	0.122	1	0.972	0.170
4. Sitting to standing	–1.863	0.117	1	0.952	0.225
8. Walking inside, with aid if needed	–1.568	0.116	1	0.936	0.261
13. Bathing	–0.872	0.115	1	0.880	0.370
5. Standing supported	–0.619	0.115	1	0.850	0.419
9. Walking outside (even ground)	0.444	0.119	1	0.662	0.714
7. Stairs	0.515	0.119	0	0.646	–1.352
11. Picking off the floor	0.940	0.122	0	0.545	–1.093
10. Walking inside, no aid	2.566	0.144	0	0.190	–0.485
12. Walking outside (uneven ground)	2.641	0.146	1	0.179	2.141
14. Up and down four steps	2.715	0.148	0	0.169	–0.450
15. Running	6.042	0.356	1	0.007	11.725

Summary

This analysis of RMI data using both traditional and Rasch psychometric analyses offers the opportunity to compare and contrast the two approaches and highlight some of the similarities and differences between them. Andrich and Styles3 suggest that the Rasch model may be considered as a refinement of, or advance on, traditional analyses. An important similarity is that both methods view the total score produced by summing the item scores as a key statistic. However, the reasons underpinning this are different: in traditional methods the total score is important because the analyses use total scores; in Rasch analysis total scores are important because a property of the Rasch model is that the total score is the sufficient statistic from which accurate estimates can be derived.

This study has demonstrated a number of refinements that result from using the Rasch model to analyse the RMI data. These refinements can be considered, separately, in terms of the construction of the scale and the measurement of people.

In Rasch analysis, the 15 items of the RMI have been located, relative to each other, on an interval-level continuum. Moreover, these estimates are independent of the distributional properties of the sample from which the estimates were made, and we have an estimate of the error associated with the location. The fact that these estimates are freed up from the sample distributional properties is a fundamental requirement if we are to determine the stability of these locations, and thus the stability of the ruler they imply, across clinically different samples. Traditional psychometric methods do not generate estimates of item locations. This does not mean we can’ t get an idea of their potential locations – this can be achieved by examining the item mean scores. However, these values are not on an interval-level scale, and are dependent on the distribution of the sample from which they were derived. Thus, inferences about the potential stability of items (even if they could be made) could not be undertaken meaningfully.

In addition to generating numerical location estimates for the items, RUMM2020 generates a graph that allows these estimates to be seen. This plot enables investigators to determine, immediately, the extent to which a set of items maps out the intended variable. The breadth of measurement, coverage across the continuum and gaps in the measurement process are explicit. This provides the empirical basis of improved measurement by, for example, plugging gaps, extending the continuum and identifying redundant items for removal. In traditional methods, item redundancy is decided based on the correlations between pairs of items and overlapping content. The correlation between two items does not tell us about their place on the continuum.

In Rasch analysis, the extent to which a set of items works together to define a single variable, their internal consistency, is determined rigorously: the fit of the observed responses to a mathematical model. In traditional methods, the internal consistency of a set of items is determined by correlational analyses: corrected item–total correlations, Cronbach’ s alpha and homogeneity coefficients. Correlations, as Thurstone pointed out nearly 80 years ago,40 are limited as ‘they constitute an acknowledgement of failure to rationalise the problem and to establish the functions that underlie the data’ .

Traditional methods and Rasch analysis came to different conclusions about the extent to which the 15 RMI items are a conformable set. Traditional methods implied the items were a cohesive set but identified item 15 (‘running’) to be poorly related to the others. Rasch analysis identified problems with a number of items that warranted further explanation. Certainly, attention needs to be paid to the wording of some items and their descriptions. Item 15 was fine, and the reasons for its poor item–total correlation are its distant location from the other items and its relative targeting to the sample. This highlights a poorly understood limitation of correlations. 122

Finally, in terms of examining items, Rasch analysis enabled a formal examination of dependency among items and differential performance of items. This was not possible with traditional methods.

Traditional and Rasch analysis provide different information about the measurement of persons and the inferences than can be made from them. The first difference is that traditional methods generate total scores and ‘measure’ people on an ordinal scale, whilst Rasch analysis generates measurements and measures people on an interval scale. Moreover, this is the same interval-level scale on which the items are located. Raw scores, which increase in successive integers, are non-linear because they have a non-linear relationship to the underlying trait (here mobility) that they seek to measure. In contrast, Rasch-derived person locations (and item locations) are linear measures because they have a linear relationship to the underlying trait that they seek to measure.

We have seen that the change in measurement implied by a 1-point change in RMI raw score varies across the range of the scale from 0.494 logits (score change in RMI 4 to 5) to 2.334 (score change in RMI 14 to 15). Thus, a change of 1 RMI raw score varies 4.7-fold across the scale range. This has critical implications for clinical trials in which accurate measurement of change underpins the inferences made. It also has implications for performing basic statistical tests on RMI raw scores. It questions the legitimacy of adding (which underpins means and SDs), subtracting (which underpins change score analysis), division and multiplication and all the statistical tests that handle data in this way. In contrast, it is legitimate to undertake these statistical analyses on Rasch-derived measurements because they are on an interval scale.

Person locations generated by a Rasch analysis are freed up from the distributional properties of the items from which they were generated. This is fundamental to examining the stability of person measurements, and thus the whole idea of equating scales on the same metric, and using different combinations of items to measure people in the knowledge that their measurements can be referred back to a common metric (item banking).

Another advantage of Rasch analysis is that it generates a standard error for each person location. In contrast, traditional analyses generate a single estimate of the standard error that is considered applicable across the whole range of the scale. The problem with a single estimate of error is that it is illogical – it is clear that the people for whom we have the least confidence about their measurement are those at the extremes of the scale range.

The availability of individualised standard errors makes measurement at the individual person level a legitimate process. This means that rating scale data from clinical trials and other studies could be analysed for individual person clinical decision making as well as group comparison study. The advantages of this are illustrated in Chapter 8. In contrast, raw scores are not considered suitable for individual person-level analysis. 107

The demonstration that variable standard errors are associated with different person location estimates is important for another reason. It demonstrates that significant change for any one individual is not simply a function of the magnitude of their change. It also depends on their location on the continuum at the measurement time points. This important fact is not accounted for in group-based analyses of change. This is discussed further in Chapter 8.

Traditional psychometric analyses do not give us information about individuals’ responses to items. Rasch analysis does, and quantifies this as the person fit statistic. This information is clinically valuable for identifying individuals for whom the pattern of responses is unlikely. Once identified, steps can be taken to diagnose why these people gave these responses. In addition, individuals identified in this manner might be excluded from the analysis as they could confound the results and inferences made.

Last, let us say a few words on missing responses to items. This occurs in many studies, although not in the RMI data from the CAMS study. As we have discussed, the traditional psychometric approach to missing item response data is to replace missing responses with a person-specific mean score. Thus, the missing item responses are replaced with the mean score of the completed items (provided that 50% of the items have been completed). This method is reputed to be psychometrically sound,36 although to our knowledge it has not been tested formally. It might be reasonable if all the items were measured at the same point on the scale (one of Likert’ s original scaling criteria6). However, as demonstrated here for the RMI, this is clearly not the case. It does not appear scientifically sound to make assumptions as to how people might have responded to items, especially as the basis for making those assumptions is weak. Rasch analysis takes a more scientific different approach to missing data. It simply uses the available data to generate a best estimate. Missing data are accounted for in terms of the confidence of the estimate – the standard error is increased.

From the above discussion, it seems clear that Rasch analysis offers substantial clinical and scientific advantages over traditional psychometric methods in the development and evaluation of rating scales, and in the analysis of rating scale data. Nevertheless, there have been a few publications that have concluded that the benefits are modest. 123 This has led a number of people to argue that traditional psychometric methods are just as good as new psychometric methods. However, these studies simply examined the impact of inferences on a single study of using Rasch transformed summed scores versus summed scores and therefore there are a number of reasons why the findings are not surprising. First, as we have seen in Figure 13, summed scores are monotonically related to the interval measurements they imply. Second, group-based and individual person-based analyses can come to different conclusions. Third, neither of these studies examined differences at the individual person level. Fourth, the impact of inferences will be study dependent; thus, findings from a single study will not generalise and it will be impossible to determine in advance the situations in which a difference might be found.

More importantly, we think that the main problem with these studies comparing Rasch with traditional scoring is that they have missed the point. As we have seen, the ability to construct interval measurements from ordinal rating scale data is one of a series of advantages that Rasch analysis offers over traditional psychometric methods. Their major role will be in the development of newer scales,119 recommended modifications of scales developed using traditional psychometric approaches and informing about the constructs being measured. 66 As rating scales are increasingly the central dependent variable on which treatment decisions are made about patients we treat, we do not feel that there can be any compromise in the efforts made to achieve rigorous measurements in clinical studies.

Overview

The aim of this chapter is to report a comprehensive evaluation of the Multiple Sclerosis Impact Scale (MSIS-29; see Appendix 2.1) using Rasch analysis, and to use this as another vehicle for comparing and contrasting the traditional and Rasch-based evaluations of rating scales. This chapter builds on the previous chapter. The RMI, the subject of Chapter 4, has items with two response categories. In contrast, the MSIS-29 has items with five response categories.

The MSIS-29 was originally developed using traditional psychometric methods. In doing so, it has fully satisfied the requirements of traditional psychometric evaluations. However, since we developed the scale, there has been increasing interest in the application of new psychometric methods (Rasch analysis and Item Response Theory). Thus, the aim of this analysis was to examine how a scale developed using traditional psychometric methods (i.e. the MSIS-29) stands up to evaluation using new psychometric methods, in this case Rasch analysis.

Although Rasch analysis provides support for the MSIS-29, it also highlighted problems with the MSIS-29 that were not detected by traditional psychometric analyses. Specifically, there were problems with the five category response options and the extent to which the items of the two subscales were conformable sets. In addition, Rasch analysis indicated how these problems may be solved to produce a more reliable, valid and responsive scale. Results also have implications for scale development. They highlight the advantages of using Rasch analysis, building scales on a strong conceptual basis; establishing the response options early in the process; and measuring over a wide range of the continuum.

The clinical problem

Rating scales are increasingly used as outcome measures in clinical studies. As such, they are the central dependent variables on which clinical decisions are made. 124 This important role necessitates that every effort be made to maximise the scientific quality of the rating scales used in clinical research and practice.

There are two main approaches to rating scale development and evaluation. The most widely used approach applies traditional methods. There is, however, increasing interest in the application of new psychometric methods. Both approaches have the same goal: to determine if it is legitimate to generate a total score by combining the integer scores from a group of items, and if that total score is reliable and valid. The two approaches differ in the evidence used to achieve that goal. Traditional psychometric analyses are based on Classical Test Theory and are embodied in the work of Likert6,29 and others. Their evidence comes mainly from analyses of correlations. New psychometric methods stem from the work of Thurstone,125 Lord and Novick,17 and Rasch. 65 Their evidence comes from checking the observed data against an a priori explicit mathematical model of how a set of items must behave to permit the summation of items to generate a reliable and valid total score. 3

New psychometric methods offer potential scientific and clinical advantages over traditional methods. From a scientific perspective they enable more sophisticated evaluations of scales. From a clinical perspective they offer, amongst other things, the ability to transform ordinal scores into interval measures, and legitimate individual person measurement. However, there are limited comparisons in the literature to determine the added value of using the new psychometric methods.

The Multiple Sclerosis Impact Scale (MSIS-29)

The MSIS-29 (see Appendix 2) is a 29-item self-report rating scale for measuring the impact of MS. It has two subscales – a 20-item physical impact scale and a nine-item psychological impact scale – and seeks to measure the impact of MS on physical and psychological functioning. All items have five response categories (‘not at all’ ; ‘a little’ ; ‘moderately’ ; ‘quite a bit’ ; ‘extremely’) that are assigned sequential integers (1, 2, 3, 4, 5). Total scores for the two scales are achieved by summing the item scores for the 20 physical items (items 1–20) and the nine psychological impact items (items 21–29). When some items have not been endorsed by an individual, a total score can still be computed provided that at least 50% of the items have been answered (i.e. 10 or more physical impact items; five or more psychological impact items). Under these circumstances investigators can replace each missing item score with the person-specific item mean score, i.e. the mean score of the items completed by that individual. This process is called imputing and is considered to be psychometrically sound. 36

The MSIS-29 was generated to be suitable for use as an MS outcome measure in appropriate clinical trials, epidemiological studies, audit and routine clinical practice. It is increasingly widely used and has been officially translated into around 20 different languages.

The MSIS-29 was developed using traditional psychometric methods for scale development. Full details of its development and evaluation are described elsewhere. 2,126 Briefly, some 3000 statements concerning the impact of MS were generated from 30 tape-recorded one-to-one patient interviews, literature review and expert opinion. These statements were examined for their content, overlap and redundancy. From the statements 141 potential scale items were written. These items were administered as a questionnaire to a large, randomly selected and geographically stratified sample of people with MS (N = 1250) from the membership database of the Multiple Sclerosis Society of Great Britain and Northern Ireland.

Data analysis focused on 129 items. This was because 12 items concerning walking were excluded as they were appropriate only to a limited number of people with MS. These 12 items formed the MS Walking Scale,127 which has since been genericised for neurological conditions. 128,129

Full details of the item reduction process, i.e. the methods used to select the final 29 items from the original 129, are given elsewhere. 126 First, 36 items were removed due to high item–item correlations and thus presumed redundancy. Next, 51 items were removed on the basis of psychometric performance (either high item-level floor/ceiling effects or poor endorsement profiles). The remaining 42 items were entered into an exploratory factor analysis. A number of possible factor solutions were examined. The two-factor solution was the most clinically and statistically appropriate, but three items were removed as they loaded similarly on both factors. The other 39 items were grouped into two scales (26 items and 13 items) whose content concerned the physical and psychological impact of MS. Refinement of the two scales using tests of item convergent and discriminate validity identified items that might confound measurement and thus were removed. The final instrument had two scales: a 20-item physical impact scale and a nine-item psychological impact scale.

The reliability and validity of the MSIS-29 were examined in a second large independent survey. We examined data quality, scaling assumptions, targeting, reliability (internal consistency and test–retest reliability) and validity (convergent and discriminate construct validity, group differences, hypothesis testing), and undertook a provisional responsiveness study. A comprehensive responsiveness study, in which the MSIS-29 was compared with a range of other scales, was also undertaken and reported. 14 Across the range of health outcomes measurement literature, it is extremely rare to find examples which include the extent of scale evaluation in all the areas we have examined in the case of the MSIS-29. Subsequent evaluations,130–133 also using traditional psychometric methods, have supported the reliability, validity and responsiveness of the scale.

Sample

MSIS-29 data from a total of 1725 people with MS were analysed. These data were generated by the two field tests of the MS Society membership databases (n = 768 and n = 712) undertaken during the development of the MSIS-292,126 and a study of its responsiveness (n = 245). 14 In this responsiveness study we had three samples of people with MS: n = 64 patients admitted for inpatient rehabilitation at the National Hospital for Neurology and Neurosurgery (NHNN) in London; n = 77 patients admitted to the NHNN for steroid treatment of MS relapses; and n = 104 people with primary progressive MS from the NHNN database. Table 14 shows the sample characteristics.

Methods

Results I: MSIS-29 physical impact subscale

Is the scale-to-sample targeting adequate for making judgements about the performance of the scale and the measurement of people?

Figure 15 shows the targeting of the patient sample to the location’ s 20 items of the physical impact subscale. Item locations range from about −1 to +1 logits. Person locations range from about −4.5 logits to +4.5 logits. Thus, the sample covers the scale well, indicating that this is a good sample for examining the performance of the scale. In contrast, the scale does not cover the full range of the sample, implying that the scale is suboptimal for measuring the people in the sample. However, each item location represents the mean of four thresholds. Figure 16 shows the targeting of the patient sample to the thresholds of the 20 items of the physical impact subscale. Item locations now spread from about −2 logits to +2 logits but still do not cover the range of person locations in the sample.

FIGURE 15.

MSIS-29 physical subscale – targeting of sample to item locations. Person–item location distribution (grouping set to interval length of 0.20, making 50 groups).

FIGURE 16.

MSIS-29 physical subscale – targeting of sample to item locations. Person–item threshold distribution (grouping set to interval length of 0.20, making 50 groups).

Has a measurement ruler been constructed successfully?

Do the item response categories work as intended?

Table 15 shows the threshold estimates and the location estimates (mean of the four threshold estimates) for the 20 items of the physical impact subscale. Nine items had reversed thresholds (1, 4, 12, 14, 15 and 17–20), i.e. the threshold estimates were not ordered sequentially. For example, for item 1, the estimate for τ3 (−0.861) is less than that for τ2 (−0.649). This means that the estimated point on the continuum at which the probability of scoring either 3 (‘moderately’) or 4 (‘quite a bit’) is the same is lower than the point on the continuum at which the probability of scoring either 2 (‘a little’) or 3 (‘moderately’) is the same. Clearly, this does not make sense and indicates that the response categories are not working as intended.

TABLE 15 - MSIS-29 physical impact subscale: item thresholds and location estimates

Location, mean of thresholds; RT, reversed threshold; X, item with reversed thresholds.

τ1, point at which probability of responding ‘not at all’ and ‘a little’ is the same.

τ2, point at which probability of responding ‘a little’ and ‘moderately’ is the same.

τ3, point at which probability of responding ‘moderately’ and ‘quite a bit’ is the same.

τ4 = point at which probability of responding ‘quite a bit’ and ‘extremely’ is the same.

Item	Location	Threshold estimates				RT
		τ1	τ2	τ3	τ4
1	–0.880	–2.165	–0.649	–0.861	0.155	X
2	0.443	–0.363	0.299	0.644	1.194
3	–0.190	–1.011	–0.319	–0.029	0.598
4	–0.427	–1.589	–0.353	–0.423	0.658	X
5	0.022	–0.983	–0.168	0.334	0.903
6	0.107	–1.632	0.183	0.313	1.565
7	0.106	–0.916	–0.112	0.101	1.352
8	–0.240	–1.248	–0.289	–0.257	0.833
9	0.764	0.033	0.366	0.702	1.955
10	0.513	–0.336	0.371	0.503	1.513
11	–0.239	–1.465	–0.149	–0.133	0.791
12	–0.091	–1.106	0.170	–0.192	0.766	X
13	0.291	–0.618	0.187	0.355	1.239
14	0.080	–0.276	0.036	–0.105	0.664	X
15	0.392	–0.587	0.406	0.299	1.450	X
16	–0.030	–0.822	–0.142	0.041	0.802
17	0.179	–0.097	0.245	0.219	0.351	X
18	–0.443	–1.906	–0.284	–0.409	0.829	X
19	–0.229	–0.725	–0.115	–0.204	0.130	X
20	–0.129	–0.892	–0.112	–0.144	0.632	X

All nine items with reversed thresholds have the same pattern of reversal, i.e. threshold τ3 is less than threshold τ2. This finding implies that people are not reliably discriminating between the three middle response categories: ‘a little’ , ‘moderately’ and ‘quite a bit’ . This consistent finding implies that there may be too many response categories or that the wording attached to the response categories is difficult for people to relate to in practice.

The category probability curve (CPC) for each item shows these relationships graphically. Figure 17 is the CPC for item 5. This graph plots the probability of a response (y-axis) against a person’ s location on the physical functioning continuum mapped out by the 20 items of the MSIS-29 physical impact subscale. Each of the four lines, labelled 0–4, represents the probability of responding to category 0–4. (Note that the RUMM2020 progam has renumbered the categories 1–5 as categories 0–4.)

FIGURE 17.

MSIS-29 physical subscale – category probability curve for item 5.

For the MSIS-29, high scores indicate more disability (the reverse of the RMI). Thus, as one moves from left to right along the x-axis, people become more physically disabled. Logically, we would expect that, as a person’ s level of physical disability increases, the probability of that person responding ‘not at all’ (the curve labelled 0 on the graph) falls and the probability of responding ‘a little’ (the curve labelled 1 on the graph) increases and becomes the most likely response category chosen. As disability increases, the probability of responding ‘a little’ increases, then decreases, and the probability of responding ‘moderately’ (the curve labelled 2 on the graph) becomes the most likely response category chosen. As the person becomes more disabled, the probability of responding ‘quite a bit’ (the curve labelled 3 on the graph) increases, becomes the most likely chosen category and is finally replaced by ‘extremely’ as the most likely response category. Thus, the response categories of item 5 are working as intended. The intersections between adjacent curves (0 and 1; 1 and 2; 2 and 3; 3 and 4) represent the points at which the probability of responding to either of a pair of adjacent response categories (e.g. 0 and 1) are the same. These points mark the four thresholds for item 5, they are ordered sequentially along the continuum (−0.983; −0.168; +0.334; +0.903), and so the response categories of item 5 are working as intended (Figure 17).

Figure 18 shows the CPC for item 1, one of the items with reversed thresholds. This graph shows that there is no point on the continuum at which response categories 2 and 3 have the highest probability of being chosen. Thus, the intersection of response categories 1 and 2 (τ2 = −0.649) is above the intersection of response category 2 (τ3 = −0.861). Hence the thresholds are disordered and the response categories are not working as intended.

FIGURE 18.

MSIS-29 physical subscale – category probability curve for item 1.

Examination of the CPCs of all the nine items with disordered thresholds shows a similar pattern to Figure 18: the category labelled 2 (‘moderately’) is never the category with the highest probability of being endorsed. Thus, people whose disability location is at the relevant range of the continuum are more likely to choose either of the two flanking categories (‘a little’ or ‘quite a bit’). It is important to recognise that this does not mean that the category has not been endorsed. Indeed, 13% of the sample (n = 206) endorsed ‘moderately’ for item 1.

Do the items map out a discernible line of increasing intensity?

Table 15 shows that the item locations (mean of four thresholds) range from −0.880 (item 1) to +0.764 (item 9), and that the item thresholds range from −2.165 (item 1 τ1) to +1.955 (item 9 τ4). Thus, the items spread out and map out a continuum. However, the spread of item locations is less than the item range for the RMI (−3.0 to +6.0).

Figure 19 shows the continuum mapped out by the 20 items. Items tend to be bunched towards the centre of the scale range. There are notable gaps in the continuum towards the extremes, indicating areas where there is suboptimal measurement and pointing to sites for improvement.

FIGURE 19.

MSIS-29 physical subscale – continuum mapped out by the 20 physical impact items.

Is the location of items along this line reasonable?

The ordering of items along the continuum mapped out by the MSIS-29 physical impact subscale makes clinical sense. The items with the most negative locations, i.e. those items that are first to become problematic when people develop disability, are ‘doing physically demanding tasks’ , ‘taking longer to do things’ and ‘problems with balance’ . Similarly, the items at the other end of continuum, which are the items that typically present a problem at greater levels of disability, are ‘difficulties using hands’ , ‘gripping things’ and ‘muscle spasms’ .

It is, however, easier to judge the suitability of item ordering when items have dichotomous response options. This is because they have a single location. In contrast, when items have multiple response options, the single location estimates are, by definition, a summary of their multiple thresholds.

Do the items work together to define a single variable?

Item–person fit residuals

Table 16 shows the fit statistics in the order the items appear on the questionnaire. Table 17 shows them ordered by fit residual magnitude. Figure 20 shows the values diagrammatically with boundary lines drawn at −2.5 and +2.5.

TABLE 16 - MSIS-29 physical impact subscale: item locations, standard errors and fit statistics [n = 1671 (adjusted to n = 500 for calculation of χ2); 10 class intervals]

Item		Location	Standard error	Fit statistics
Number	Label			Fit residual	χ2	χ2 probability
1	Do physically demanding tasks	–0.880	0.029	–0.734	4.468	0.878
2	Grip things tightly	0.443	0.025	3.714	4.095	0.905
3	Carry things	–0.190	0.026	–1.851	5.795	0.760
4	Problems with your balance	–0.427	0.027	1.501	4.014	0.911
5	Difficulties moving about indoors	0.022	0.026	–7.023	26.168	0.002
6	Being clumsy	0.107	0.027	–0.307	5.478	0.791
7	Stiffness	0.106	0.026	6.002	10.882	0.284
8	Heavy arms and/or legs	–0.240	0.026	5.728	7.816	0.553
9	Tremor of your arms or legs	0.764	0.026	5.789	12.350	0.194
10	Spasms in your limbs	0.513	0.025	4.901	17.612	0.040
11	Your body not doing what you want it to do	–0.239	0.026	–5.300	15.268	0.084
12	Having to depend on others to do things for you	–0.091	0.025	–8.122	28.297	0.001
13	Limitations in social and leisure activities at home	0.291	0.025	0.657	3.981	0.913
14	Being stuck at home more than you would like	0.080	0.024	1.358	2.409	0.983
15	Difficulties using your hands in everyday tasks	0.392	0.025	–1.842	6.086	0.731
16	Having to cut down time spent on work/daily activities	–0.030	0.025	3.084	9.160	0.423
17	Problems using transport	0.179	0.023	1.079	3.807	0.924
18	Taking longer to do things	–0.443	0.027	–7.887	29.895	0.000
19	Difficulty doing things spontaneously	–0.229	0.024	–0.880	6.074	0.733
20	Needing to go to the toilet urgently	–0.129	0.025	15.845	73.537	0.000

TABLE 17 - MSIS-29 physical impact subscale: items ordered by increasing item–person fit residual [n = 1671 (adjusted to n = 500 for calculation of χ2); 10 class intervals]

Item		Location	Standard error	Fit statistics
Number	Label			Fit residual	χ2	χ2 probability
12	Having to depend on others to do things for you	–0.091	0.025	–8.122	28.297	0.001
18	Taking longer to do things	–0.443	0.027	–7.887	29.895	0.000
5	Difficulties moving about indoors	0.022	0.026	–7.023	26.168	0.002
11	Your body not doing what you want it to do	–0.239	0.026	–5.300	15.268	0.084
3	Carry things	–0.190	0.026	–1.851	5.795	0.760
15	Difficulties using your hands in everyday tasks	0.392	0.025	–1.842	6.086	0.731
19	Difficulty doing things spontaneously	–0.229	0.024	–0.880	6.074	0.733
1	Do physically demanding tasks	–0.880	0.029	–0.734	4.468	0.878
6	Being clumsy	0.107	0.027	–0.307	5.478	0.791
13	Limitations in social and leisure activities at home	0.291	0.025	0.657	3.981	0.913
17	Problems using transport	0.179	0.023	1.079	3.807	0.924
14	Being stuck at home more than you would like	0.080	0.024	1.358	2.409	0.983
4	Problems with your balance	–0.427	0.027	1.501	4.014	0.911
16	Having to cut down time spent on work/daily activities	–0.030	0.025	3.084	9.160	0.423
2	Grip things tightly	0.443	0.025	3.714	4.095	0.905
10	Spasms in your limbs	0.513	0.025	4.901	17.612	0.040
8	Heavy arms and/or legs	–0.240	0.026	5.728	7.816	0.553
9	Tremor of your arms or legs	0.764	0.026	5.789	12.350	0.194
7	Stiffness	0.106	0.026	6.002	10.882	0.284
20	Needing to go to the toilet urgently	–0.129	0.025	15.845	73.537	0.000

FIGURE 20.

MSIS-29 physical subscale – plot of item–person fit residuals against location.

Nine items fall within the recommended range (1, 3, 4, 6, 13, 14, 15, 17 and 19). A further two items (2 and 16) are near the boundary lines. One item (item 20, ‘needing to go to the toilet urgently’) is very misfitting, and eight items are notably misfitting.

Chi-squared values and probabilities

Table 16 shows the fit statistics in item sequence order. Table 18 shows them ordered by chi-squared values. Figure 21 shows the values diagrammatically.

TABLE 18 - MSIS-29 physical impact subscale: items ordered by increasing chi-squared value [n = 1671 (adjusted to n = 500 for calculation of χ2); 10 class intervals]

Item		Location	Standard error	Fit statistics
Number	Label			Fit residual	χ2	χ2 probability
14	Being stuck at home more than you would like	0.080	0.024	1.358	2.409	0.983
17	Problems using transport	0.179	0.023	1.079	3.807	0.924
13	Limitations in social and leisure activities at home	0.291	0.025	0.657	3.981	0.913
4	Problems with your balance	–0.427	0.027	1.501	4.014	0.911
2	Grip things tightly	0.443	0.025	3.714	4.095	0.905
1	Do physically demanding tasks	–0.880	0.029	–0.734	4.468	0.878
6	Being clumsy	0.107	0.027	–0.307	5.478	0.791
3	Carry things	–0.190	0.026	–1.851	5.795	0.760
19	Difficulty doing things spontaneously	–0.229	0.024	–0.880	6.074	0.733
15	Difficulties using your hands in everyday tasks	0.392	0.025	–1.842	6.086	0.731
8	Heavy arms and/or legs	–0.240	0.026	5.728	7.816	0.553
16	Having to cut down time spent on work/daily activities	–0.030	0.025	3.084	9.160	0.423
7	Stiffness	0.106	0.026	6.002	10.882	0.284
9	Tremor of your arms or legs	0.764	0.026	5.789	12.350	0.194
11	Your body not doing what you want it to do	–0.239	0.026	–5.300	15.268	0.084
10	Spasms in your limbs	0.513	0.025	4.901	17.612	0.040
5	Difficulties moving about indoors	0.022	0.026	–7.023	26.168	0.002
12	Having to depend on others to do things for you	–0.091	0.025	–8.122	28.297	0.001
18	Taking longer to do things	–0.443	0.027	–7.887	29.895	0.000
20	Needing to go to the toilet urgently	–0.129	0.025	15.845	73.537	0.000

Figure 21 shows that item 20 misfits substantially compared with the rest. Three further items (5, 12 and 18) are more misfitting than the main group. The chi-squared probability for these four items is < 0.01, indicating that the difference between observed scores and expected values is large relative to chance.

The remaining 16 items have more similar-level chi-squared values and Table 18 shows that there is a gradual increase in chi-squared value from 2.4 (item 14) to 17.6 (item 10) before there is a notable step increase in value (item 5; chi-squared value = 26.2).

Item characteristic curves

Figure 22 shows the ICCs for the four items that ‘failed’ both the item–person fit residual and the item–trait chi-squared value tests of fit. All four items show reasonable adherence between the curve defined by the 10 class intervals (black dots) representing the observed data and the ICC curve representing the expected values derived from the Rasch model. Figure 22 implies less concern about the degree of misfit of these items than is implied by the values reported in Tables 16–18 and by Figures 20 and 21.

FIGURE 21.

MSIS-29 physical subscale – plot of chi-squared values against location.

FIGURE 22.

MSIS-29 physical subscale – item characteristic curves for items 20, 18, 12 and 5.

Figures 23 and 24 show the ICCs for the six items (2, 7, 8, 9, 10 and 11) for which fit residuals were outside the recommended range (−2.5 to +2.5). For all these items the visual discrepancy between observed scores and expected values across 10 class intervals is small. For comparison, Figure 25 shows the ICCs for two items (1 and 17) that passed both the item–person fit residual and item–trait chi-squared tests of fit.

FIGURE 23.

MSIS-29 physical subscale – item characteristic curves for items 7, 9, 8 and 11.

FIGURE 24.

MSIS-29 physical subscale – item characteristic curves for items 10 and 2.

FIGURE 25.

MSIS-29 physical subscale – item characteristic curves for items 1 and 17.

Note that Table 19 shows the locations, errors and fit statistics with the sample not adjusted to n = 500 to demonstrate that only the chi-squared values and chi-squared probabilities are affected.

TABLE 19 - MSIS-29 physical impact subscale: item locations, standard errors and fit statistics [n = 1671 (sample size not adjusted for χ2 calculation); 10 class intervals]

Item		Location	Standard error	Fit statistics
Number	Label			Fit residual	χ2	χ2 probability
1	Do physically demanding tasks	–0.880	0.029	–0.734	14.931	0.093
2	Grip things tightly	0.443	0.025	3.714	13.684	0.134
3	Carry things	–0.190	0.026	–1.851	19.366	0.022
4	Problems with your balance	–0.427	0.027	1.501	13.414	0.145
5	Difficulties moving about indoors	0.022	0.026	–7.023	87.452	0.000
6	Being clumsy	0.107	0.027	–0.307	18.306	0.032
7	Stiffness	0.106	0.026	6.002	36.369	0.000
8	Heavy arms and/or legs	–0.240	0.026	5.728	26.120	0.002
9	Tremor of your arms or legs	0.764	0.026	5.789	41.273	0.000
10	Spasms in your limbs	0.513	0.025	4.901	58.860	0.000
11	Your body not doing what you want it to do	–0.239	0.026	–5.300	51.027	0.000
12	Having to depend on others to do things for you	–0.091	0.025	–8.122	94.570	0.000
13	Limitations in social and leisure activities at home	0.291	0.025	0.657	13.306	0.149
14	Being stuck at home more than you would like	0.080	0.024	1.358	8.052	0.529
15	Difficulties using your hands in everyday tasks	0.392	0.025	–1.842	20.339	0.016
16	Having to cut down time spent on work/daily activities	–0.030	0.025	3.084	30.614	0.000
17	Problems using transport	0.179	0.023	1.079	12.722	0.176
18	Taking longer to do things	–0.443	0.027	–7.887	99.910	0.000
19	Difficulty doing things spontaneously	–0.229	0.024	–0.880	20.298	0.016
20	Needing to go to the toilet urgently	–0.129	0.025	15.845	245.762	0.000

Does the response to one item directly influence the response to another?

In general, the correlations between the residuals were low. Only four of the 190 correlations exceeded 0.30, and none exceeded 0.40. The pairs of items whose residual correlations exceeded 0.30, and their values were:

1. Items 2 and 3 = 0.32

2. Items 2 and 15 = 0.37

3. Items 9 and 10 = 0.38

4. Items 13 and 14 = 0.33

Are the locations of the items stable across clinically important groups?

We examined differential item functioning (DIF) in relation to five clinically important subgroupings: sex (men and women); mobility level (unaided; with an aid; in a wheelchair); MS type (relapsing–remitting MS; secondary progressive MS; primary progressive MS); educational level (higher degree/qualification or not); sample (rehabilitation; steroids; first postal survey; second postal survey).

In the context of these subgroupings, DIF was only demonstrated in one circumstance (mobility level). Here, three items (5, 12 and 20) demonstrated statistically significant DIF. However, a number of class intervals had very small numbers (approximately 15). Nevertheless, re-evaluation of DIF using two class intervals produced the same results.

Have the people in the sample been measured successfully?

Are the persons in the sample separated along the line defined by the items?

Figures 15 and 16 show the distribution of person locations compared with the item locations (see Figure 15) and item thresholds (see Figure 16). It is clear from these two figures that the sample locations are spread across the continuum. This is reflected in the PSI, which is 0.955, indicating that the 20 physical impact items are able to distinguish reliably between responders on the trait they measure.

Figure 26 is a plot of raw scores against the interval measures they imply. The relationship is S-shaped (an ogive) rather than linear (a straight linear). This indicates that equal changes in one metric are not associated with equal changes in the other. The correlation between raw scores and interval measures is 0.952.

FIGURE 26.

MSIS-29 physical subscale – plot of raw scores against interval measures.

Table 20 gives the interval-level location values that correspond with each raw score. Note that a change from 0 to 1 raw score point corresponds to a change in interval measure of 0.81 logits. A change from 42 to 43 raw score points corresponds to a change in interval measure of 0.03 logits. Thus, the implication of a 1-point change in raw score varies up to 27 times across the range of the scale.

TABLE 20 - MSIS-29 physical impact subscale: interval-level locations implied by each ordinal-level raw score

Note that the measurements for people at the extremes of the scale range (floor and ceiling) are extrapolation based on actual estimates. For these people no person–item fit residual can be computed.

Raw score	Interval measure	Standard error
0	–4.490a	1.180
1	–3.684	0.840
2	–3.144	0.650
3	–2.783	0.560
4	–2.510	0.490
5	–2.291	0.450
6	–2.108	0.410
7	–1.950	0.380
8	–1.813	0.360
9	–1.691	0.340
10	–1.582	0.320
11	–1.482	0.310
12	–1.391	0.300
13	–1.307	0.290
14	–1.229	0.280
15	–1.156	0.270
16	–1.088	0.260
17	–1.023	0.250
18	–0.961	0.250
19	–0.903	0.240
20	–0.847	0.240
21	–0.793	0.230
22	–0.741	0.230
23	–0.691	0.220
24	–0.643	0.220
25	–0.596	0.220
26	–0.551	0.210
27	–0.507	0.210
28	–0.463	0.210
29	–0.421	0.210
30	–0.380	0.210
31	–0.339	0.200
32	–0.298	0.200
33	–0.259	0.200
34	–0.220	0.200
35	–0.181	0.200
36	–0.142	0.200
37	–0.104	0.200
38	–0.066	0.200
39	–0.028	0.200
40	0.010	0.200
41	0.047	0.200
42	0.085	0.200
43	0.123	0.200
44	0.161	0.200
45	0.199	0.200
46	0.238	0.200
47	0.277	0.200
48	0.316	0.200
49	0.356	0.200
50	0.397	0.200
51	0.438	0.210
52	0.480	0.210
53	0.523	0.210
54	0.566	0.210
55	0.611	0.220
56	0.657	0.220
57	0.705	0.220
58	0.754	0.230
59	0.805	0.230
60	0.857	0.230
61	0.912	0.240
62	0.970	0.240
63	1.030	0.250
64	1.094	0.260
65	1.161	0.260
66	1.232	0.270
67	1.308	0.280
68	1.390	0.290
69	1.478	0.300
70	1.575	0.320
71	1.681	0.330
72	1.799	0.350
73	1.931	0.380
74	2.083	0.400
75	2.260	0.440
76	2.471	0.480
77	2.735	0.550
78	3.086	0.650
79	3.613	0.830
80	4.400a	1.170

Table 20 also includes the standard errors associated with each location. These vary across the range of the scale, being largest at the extremes and smallest in the centre of the scale range. Figure 27 shows the relationship between standard error and location on the continuum. The relationship is U-shaped and shows how much the precision of measurement falls as locations move away from the centre of the scale. Figure 28 also shows this relationship, superimposed on the sample and item distribution. Essentially, Table 20 and Figures 27 and 28 show that good measurement occurs only at the centre of the scale range.

FIGURE 27.

MSIS-29 physical subscale – plot of standard error against location.

FIGURE 28.

MSIS-29 physical subscale – information from scale relative to sample. Person–item threshold distribution (grouping set to interval length of 0.20, making 50 groups).

It is important to note that the raw scores, interval locations and standard errors shown in Table 20 are for complete data. By complete data we mean people who have endorsed a response category for all 20 items. The location estimates for people who have not completed all 20 items will depend on the items they endorsed and the total score they achieved. This could be overcome by imputing for missing data using a person-specific mean score (as discussed earlier).

Another point to note is that the location estimate and standard error corresponding to each raw score is the same irrespective of the pattern of item responses (provided that all 20 items have been endorsed). This does not mean that the pattern of responses is irrelevant. On the contrary, in Rasch analysis differences in the pattern of item responses are reflected in the person–item fit statistic (discussed below).

Finally, as noted before, the range of the total scores in Table 20 is 0–80. The response categories on the questionnaire are labelled 1–5, and thus the total score for the 20 physical impact items ranges from 20 to100. The reason for the difference is that in the RUMM2020 program the first item response option is labelled 0. To convert from questionnaire-determined total score to RUMM2020-determined total score simply subtract 20.

Do individual placements on the variable make sense?

The placements of individuals on the physical function continuum mapped out by the items was largely consistent with expectation. For example, people who walked without an aid had worse physical function than people who used a walking aid, and people who used a walking aid had worse physical function than people who used a wheelchair.

How valid is each person’ s measurement?

Person–item fit residuals indicate the extent to which the responses of a specific individual are consistent with expectation based on the Rasch model. They are analogous to the item–person fit residuals that indicate the extent to which the responses to a specific item are consistent with expectation based on the Rasch model. It is recommended that person–item fit residuals lie in the range −2.5 to +2.5.

In our sample of 1723 people, person–item fit residuals could be computed for 1671, as they could not be computed for the 52 people who scored at the extremes. Values were outside the recommended range for 189 people (11% of the sample). These exceeded −3.0 to +3.0 in only 16 people (1%).

Results II: MSIS-29 psychological impact subscale

Is the scale-to-sample targeting adequate for making judgements about the performance of the scale and the measurement of people?

Figure 29 shows the targeting of the item locations to the sample. Figure 30 shows the targeting of the item thresholds to the sample. The sample covers the scale well, implying that this is a good sample in which to study the scale. However, the scale does not cover the sample well, implying that the scale is suboptimal for measuring the people. This is discussed further below.

FIGURE 29.

MSIS-29 psychological subscale – targeting of sample to item locations. Person–item location distribution (grouping set to interval length of 0.20, making 40 groups).

FIGURE 30.

MSIS-29 psychological subscale – targeting of sample to item thresholds. Person–item threshold distribution (grouping set to interval length of 0.20, making 40 groups).

Has a measurement ruler been constructed successfully?

Do the item response categories work as intended?

Only one item (item 22) has reversed thresholds, indicating that the item scoring function for this item was not working as anticipated. Figure 31 shows the CPC for this item. However, a close look at the CPCs for the other eight items, and their threshold values in Table 21, indicates that thresholds τ3 and τ4 for all items occur at very similar locations on the continuum. Figure 32 shows the CPCs for items 24–27. As with the physical subscale items, this finding implies that there may be too many response options in the psychological subscale items.

FIGURE 31.

MSIS-29 psychological subscale – category probability curve for item 22.

TABLE 21 - MSIS-29 psychological impact subscale: item thresholds and location estimates

Location, mean of thresholds; RT, reversed threshold; X, item with reversed thresholds.

τ1, point at which probability of responding ‘not at all’ and ‘a little’ is the same.

τ2, point at which probability of responding ‘a little’ and ‘moderately’ is the same.

τ3, point at which probability of responding ‘moderately’ and ‘quite a bit’ is the same.

τ4, point at which probability of responding ‘quite a bit’ and ‘extremely’ is the same.

Item	Location	Threshold estimates				RT
		τ1	τ2	τ3	τ4
21	0.097	–1.175	0.021	0.356	1.185
22	0.164	–0.579	0.178	0.001	1.055	X
23	–0.476	–1.623	–0.594	–0.249	0.561
24	–0.031	–1.323	0.105	0.107	0.986
25	0.053	–1.361	–0.077	0.157	1.495
26	–0.024	–1.452	0.019	0.144	1.194
27	0.036	–1.323	0.033	0.170	1.263
28	0.008	–0.853	–0.088	–0.058	1.029
29	0.173	–0.922	0.243	0.262	1.110

FIGURE 32.

MSIS-29 psychological subscale – category probability curve for items 24, 25, 26 and 27.

Do the items map out a discernible line of increasing intensity?

Table 21 shows that item locations range from −0.48 (‘feeling mentally fatigued’) to +0.17 (‘feeling depressed’), less than one logit, while item thresholds vary from −1.62 to +1.50. Thus, although the items spread out they do not spread to define a wide continuum. Figure 33 shows the psychological impact continuum mapped out by the nine items. There are clear gaps and there is bunching around the centre of the scale range. Figure 30, the plot of item thresholds, shows 10 of a total of 36 thresholds lying between 0.0 and 0.2 logits, a very narrow range.

FIGURE 33.

MSIS-29 psychological subscale – continuum mapped out by the nine psychological impact items.

Is the location of items along this line reasonable?

Table 22 shows the nine items in location order. The continuum implies that the first of these nine problems to be manifest as a person moves from left to right down the psychological impact continuum is likely to be ‘feeling mentally fatigued’ . As the psychological impact becomes greater there are likely to be problems with concentrating and anxiety. Last is the likelihood of a perception of depression.

TABLE 22 - MSIS-29 psychological impact subscale: item locations, standard errors and fit statistics ordered by increasing item location [n = 1661 (adjusted to n = 500 for calculation of χ2); 10 class intervals]

Item				Fit statistics
Number	Label	Location	Standard error	Fit residual	χ2	χ2 probability
23	Feeling mentally fatigued	–0.476	0.026	0.916	4.813	0.850
24	Worries about your MS	–0.031	0.026	1.429	2.111	0.990
26	Feeling irritable, impatient or short-tempered	–0.024	0.027	–0.686	3.506	0.941
28	Lack of confidence	0.008	0.025	–1.698	6.669	0.672
27	Problems concentrating	0.036	0.026	2.380	3.380	0.947
25	Feeling anxious or tense	0.053	0.027	–7.488	31.894	0.000
21	Feeling unwell	0.097	0.027	–0.711	4.185	0.899
22	Problems sleeping	0.164	0.025	9.602	40.791	0.000
29	Feeling depressed	0.173	0.026	–4.993	17.991	0.035

Do the items work together to define a single variable?

Item–person fit residuals

Table 23 shows the item–person fit residuals. Figure 34 shows these values diagrammatically. Three items lie outside the recommended range; one item (item 22) misfits substantially more than the other two (items 29 and 25).

TABLE 23 - MSIS-29 psychological impact subscale – item locations, standard errors and fit statistics [n = 1661 (adjusted to n = 500 for calculation of χ2); 10 class intervals]

Item				Fit statistics
Number	Label	Location	Standard error	Fit residual	χ2	χ2 probability
21	Feeling unwell	0.097	0.027	–0.711	4.185	0.899
22	Problems sleeping	0.164	0.025	9.602	40.791	0.000
23	Feeling mentally fatigued	–0.476	0.026	0.916	4.813	0.850
24	Worries about your MS	–0.031	0.026	1.429	2.111	0.990
25	Feeling anxious or tense	0.053	0.027	–7.488	31.894	0.000
26	Feeling irritable, impatient or short-tempered	–0.024	0.027	–0.686	3.506	0.941
27	Problems concentrating	0.036	0.026	2.380	3.380	0.947
28	Lack of confidence	0.008	0.025	–1.698	6.669	0.672
29	Feeling depressed	0.173	0.026	–4.993	17.991	0.035

FIGURE 34.

MSIS-29 psychological subscale – plot of item–person fit residuals against location.

Item–trait chi-squared values

Table 23 shows the chi-squared values and Figure 35 plots these values diagrammatically. The same three items with misfitting fit residuals have more extreme chi-squared values relative to the others. For two items (items 22 and 25) the chi-squared values are statistically significant, indicating that the discrepancies between observed scores and expected values are larger than expected by chance.

FIGURE 35.

MSIS-29 psychological subscale – plot of chi-squared values against location.

Item characteristic curves

All ICCs were examined. Figure 36 shows the ICCs for the three worst-fitting items (22, 25 and 29). Although the fit statistics raise concerns about these three items, the curve defined by the observed scores has an increasing monotonic relationship with location on the continuum and follows the expected values fairly well. These findings imply less concern about the extent to which these items misfit.

FIGURE 36.

MSIS-29 psychological subscale – item characteristic curves for items 22, 25 and 29.

Does the response to one item directly influence the response to another?

Correlations between residuals were low. Only two of the 36 correlations exceeded 0.30, and none exceeded 0.40. The pairs of items whose residual correlations exceeded 0.30, and their values, were:

1. Items 23 and 24 = 0.33

2. Items 24 and 27 = 0.31

Are the locations of the items stable across clinically important groups?

DIF was tested for sex (men and women); mobility level (unaided; with an aid; in a wheelchair); MS type (relapsing–remitting MS; secondary progressive MS; primary progressive MS); educational level (higher degree/qualification or not); and sample (rehabilitation; steroids; first postal survey; second postal survey). No items demonstrated differential item functioning for age, gender, sample or mobility level. Note that DIF is discussed at length in Chapter 6.

Have the people in the sample been measured successfully?

Are the persons in the sample separated along the line defined by the items?

Figure 29 demonstrates that the person locations in this sample ranged from about −4.0 to +4.0 logits. Thus, despite the limited range of item locations, the scale seems to detect important differences. The PSI is 0.892, indicating that the nine-item scale is able to distinguish reliably between the responders on the trait that it measures.

Table 24 shows the interval-level locations implied by each psychological subscale total score. Figure 37 plots the values. The correlation is 0.965, but the graph is S-shaped. Table 24 indicates that a change of 1 raw score point at the scale extreme implies a change in location of 0.80 logits, whereas in the centre of the scale a change of 1 raw score point implies a change of 0.08 logits. Thus, a change of 1 point implies a variable change in interval-level location of up to 10-fold depending on where in the scale range the change occurs.

TABLE 24 - MSIS-29 psychological impact subscale: interval-level locations implied by each ordinal-level raw score

Note that the measurements for people at the extremes of the scale range (floor and ceiling) are extrapolation based on actual estimates. For these people no person–item fit residual can be computed.

Raw score	Interval measure	Standard error
0	–3.810a	1.190
1	–2.990	0.860
2	–2.430	0.670
3	–2.050	0.570
4	–1.760	0.510
5	–1.530	0.460
6	–1.330	0.430
7	–1.160	0.400
8	–1.010	0.380
9	–0.880	0.360
10	–0.750	0.350
11	–0.640	0.340
12	–0.530	0.330
13	–0.430	0.320
14	–0.330	0.310
15	–0.240	0.310
16	–0.150	0.310
17	–0.060	0.300
18	0.020	0.300
19	0.110	0.300
20	0.200	0.300
21	0.290	0.310
22	0.380	0.310
23	0.470	0.310
24	0.570	0.320
25	0.670	0.330
26	0.780	0.340
27	0.890	0.350
28	1.020	0.370
29	1.160	0.390
30	1.320	0.420
31	1.500	0.450
32	1.720	0.490
33	1.990	0.560
34	2.350	0.650
35	2.890	0.840
36	3.700a	1.180

FIGURE 37.

MSIS-29 psychological subscale – plot of raw scores against interval measures.

Table 24 also reports the standard error associated with each person location estimate. These vary fourfold across the range of the scale and are greater at the extremes. Figure 38 plots the standard error against the location. This shows that the scale is at its most precise in the central range. Figure 39 plots the information function for the scale in relation to the study sample and the item thresholds. Many people lie outside the range in which this scale provides precise measurements.

FIGURE 38.

MSIS-29 psychological subscale – plot of standard error against location.

FIGURE 39.

MSIS-29 psychological subscale – plot of information from scale relative to sample. Person–item threshold distribution (grouping set to interval length of 0.20, making 40 groups).

Do individual placements on the variable make sense?

How valid is each person’ s measurement?

A total of 1716 people were measured in the sample. Of these, 50 scored at the floor (n = 34) or ceiling (n = 16) and did not have person–item fit residual estimates. Of the 1666 people who had estimates of their person–item fit residuals, 116 people (7%) had values that were outside the range −2.5 to +2.5, and 83 (5% of the sample) had fit residuals outside the range −3.0 to +3.0. These findings indicate that 93% of people gave responses that were consistent with expectation.

Summary

The aim of this chapter was to examine, with new psychometric methods, a scale that had been developed using traditional psychometric methods. The purpose of this examination was to compare and contrast the two approaches, to determine the added value (if any) of using a more sophisticated method, and whether any advantages justified the additional expertise required to undertake such analysis.

Implications for the MSIS-29

The MSIS-29 was developed using traditional psychometric methods. Those of us involved in its development would like to think that we applied those methods rigorously. Others think so. 135 We followed the well-established three-stage approach: item generation; item reduction and scale formation; and scale validation. People with MS were intimately involved at all stages and the study was in collaboration with the MS Society of Great Britain and Northern Ireland. To generate a pool of items, we interviewed people with MS, representing the full range of the condition, until redundancy, canvassed expert opinion from a range of health professionals and undertook a comprehensive literature review. This process generated in the region of 3000 statements concerning the health impact of MS. A preliminary questionnaire was prepared from these statements, and pre-tested in a sample of people with MS, before being sent to a large sample of people with MS across the UK.

Many approaches have been used to reduce a pool of items to form a rating scale, but there are no consensus guidelines. On account of this, we identified the many criteria used by others, and applied these whenever possible. The result was the 29-item Multiple Sclerosis Impact Scale (MSIS-29).

The MSIS-29 was validated, comprehensively, in a large, independent sample of people with MS. A range of other widely used self-report scales were used as validation instruments. The MSIS-29 satisfied all criteria that we tested, and outperformed competing scales. Others have found similar results.

Rasch analysis of the MSIS-29 provided evidence to support the findings of the traditional psychometric evaluations. However, Rasch analysis also demonstrated important limitations in the scale and in traditional psychometric methods, and provoked thinking into the constructs measured and the whole process of scale development.

Evidence in support of the measurement properties of the MSIS-29 came from a demonstration that the items of both subscales map out continua of increasing intensity; are located along those continua in a clinically sensible order; work reasonably well together to define single variables; consist of items that are locally independent; and do not exhibit differential item functioning. Further support for the use of the subscales came from the findings that both subscales were able to separate the sample reliably and that people’ s patterns of responses were consistent with expectation.

However, Rasch analysis detected important limitations of the MSIS-29 which were not identified by traditional psychometric methods. It detected that the five-category item scoring function did not work as intended for nine items in the physical subscale and one item in the psychological subscale. This implied that these items had too many response options. In fact, when the CPCs and the threshold values were examined for the items with ordered thresholds, there was good evidence that all items would benefit from fewer response categories. This can be seen quite clearly in the threshold maps, which show the region of the continuum represented by each item response option for items with ordered thresholds (items with disordered thresholds are not shown). Figure 40 shows the threshold maps for the two MSIS-29 subscales, and demonstrates that the area of the continuum represented by response label 2 (= ‘moderately’ on the questionnaire) is often very small. Because of this finding we undertook a post hoc rescoring of the items by collapsing adjacent categories (so that all items had four response categories), as suggested by each CPC. Re-analysis of the data demonstrated that all thresholds were now correctly ordered. The rest of the analyses were similar. Clearly, this needs empirical testing as collapsing categories makes assumptions about how people would respond.

FIGURE 40.

MSIS-29 subscales – threshold maps. *Reversed thresholds. Key: 0, response category labelled 0; 1, response category labelled 1; 2, response category labelled 2; 3, response category labelled 3; 4, response category labelled 4

A second limitation of the MSIS-29 detected by Rasch analysis, and applicable to both subscales but especially the psychological subscale, was the range and spread of the item locations and thresholds. Neither scale mapped a wide variable. Both scales demonstrated bunching of item thresholds and gaps in their continua. The implication of these findings is that measurement is suboptimal and could be improved. It is also important to note that these findings are a function of traditional psychometric methods. Correlation-based analyses (e.g. item–total correlations, item–item correlations, alphas), which are the mainstay underpinning item selection, tend to identify items in the middle of the scale range as superior. A number of difficulties arise from packing items in the central region of the scale range. One problem is that the scale is only a precise measure over a limited range. This is demonstrated by the subscale information functions (see Figures 28 and 39), the plots of standard errors against location (see Figures 27 and 38) and the fact that both scales only provide precise measurement to a proportion of the study sample. A second problem is that scales could become overprecise in the centre relative to the extreme. Thus, the relationship between raw scores and interval measures becomes increasingly S-shaped, and thus increasingly non-linear, so that the ‘real’ implications of raw score changes and differences becomes increasingly distorted. We have seen in this study that the change in interval-level measurement implied by a fixed change in raw score varies up to 27-fold.

A third limitation of the MSIS-29, detected by Rasch analysis, relates to the extent to which the items work together to define a single variable. Both subscales, but especially the physical subscale, demonstrated notable misfit on the two numerical indicators of fit of the data to the Rasch model (item–person fit residuals; item–trait chi-squared values). The third graphical indicator of fit implied that this misfit was less of a concern. These findings could be interpreted in two ways. On the one hand, we could favour the graphical indicator, play down the numerical values and argue that the items in each subscale scale work well together to define a single variable. On the other hand, we could be more circumspect and try to diagnose why an instrument that passes all traditional tests of validity demonstrates notable misfit on some – but not all – tests of fit. The second approach is that favoured by leading Rasch analysts,66,74,136 and in line with Kuhnian theory, which argues that the role of measurement is to highlight anomalies for further investigation. 137,138

In keeping with this approach, the results of the test of fit provoked a careful examination of the items, a consideration of the scale development process and an explanation for the findings. For example, the 20 items of the physical impact subscale can be grouped into three themes: activity limitations; symptoms causing physical limitations; and limitations in social functioning. The amalgamation of these related but different subconstructs into a single set operationalises a broad physical functioning variable and would explain why there are some problems with fit. Why did this happen? Because the scale development process used a factor analysis which groups items that are related but distinct from other groups of items. Moreover, as factor analysis is based on correlations, it will tend to bring together items that measure at a similar point on the scale, irrespective of their content.

These findings have led us to revise the MSIS-29, which is now under evaluation. (MSIS-29v2; Appendix 2.2)

Overview

This chapter uses the examination of test–retest reproducibility to illustrate similarities and differences between traditional and Rasch-based psychometric evaluations. Test–retest reproducibility refers simply to the stability of a measuring instrument (i.e. it is the ability of the measure to produce stable scores over a given period of time in which the respondent’ s condition of interest is assumed to have remained the same). On the surface, the assessment of test–retest reproducibility seems straightforward. As we will see, it is not straightforward, because of an interaction between scale stability and person stability. This chapter demonstrates how Rasch analysis is able to dissect the different components of this interaction to achieve a comprehensive understanding of item and person stability and change, whereas traditional psychometric methods are unable to do so.

The test–retest reproducibility of the MSIS-29 is shown to be good. However, that is not the main point of this chapter. Its primary goal is to use the examination of test–retest reproducibility of the MSIS-29 as a vehicle for introducing, explaining, demonstrating and discussing two important aspects of a Rasch analysis. The first of these aspects is the application of ‘racking’ and ‘stacking’ data designs. These refer to different arrangements (set-ups) of rating scale data to enable different questions to be answered by different analyses. The second aspect is the concept of differential item functioning (DIF). This is the extent to which the functioning of an item differs across different circumstances. This has been mentioned, but not actively discussed, in previous chapters.

The clinical problem

On the surface, the approach to the assessment of consistency of two reports (i.e. two completions of a questionnaire) appears straightforward; simply administer a rating scale (e.g. MSIS-29) twice to a cohort of people and determine the agreement between their total scores at the two time points. Unfortunately, this oversimplifies a complex situation because it confounds two related but independent questions. First, does the scale work in the same way internally on both occasions? Second, if the scale does work in the same way on both occasions, do the people generate equivalent measurements at the two time points? Andrich has termed these as questions of equivalence in kind and equivalence in degree. 97

It is easy to appreciate that the same rater may generate different ratings at two time points using the same rating scale. It is more difficult to appreciate that these differences may take two forms that need to be separated: first, a lack of internal stability (invariance); and, second, a lack of average measurement equivalence. The reason for this is that when rating scales are used to quantify variables, measurements are constructed from the responses of persons to a set of items. As such, there is an interaction between raters and items that must be disentangled into its component parts.

Simple indicators of agreement between total scores, even if complemented by a study of agreement between item-level scores, are unable to answer these two questions of equivalence of kind and degree independently. The reason lies in that a feature of traditional psychometric evaluations is that scale performance is dependent on the sample, which, to confound matters further, is often assumed to be normally distributed. More correctly, the performance of the scale and its items is dependent on the distribution of disability in the sample in which it was examined. Likewise, the measurement of people is dependent on the distribution of disability measured by the scale items. Thus, the performance of the scale cannot be divorced from the sample and the measurement of people cannot be divorced from the scale. Equivalence of kind and equivalence of degree cannot be separated.

Rasch analysis can overcome this problem and enable equivalence of kind and equivalence of degree to be studied independently and legitimately. This is because one mathematical property of the Rasch model is the separation of item and person parameters. To recap, the estimates of the item locations are independent of the sampling distribution of the people in whom they are studied. Likewise, estimates of person locations are independent of the sampling distribution of the items on which they were measured.

The Rasch model is able to estimate item and person locations separately because the model arises from the requirement of invariance (stability) of the operation of the items across different groups and across the quantitative trait. Therefore, checks on the fit of the data to the model are checks on the invariance of the operation of the items, i.e. a check on any difference in kind of the ratings such as difference among raters at two time points (i.e. test–retest reproducibility). This check is commonly known as a check on DIF. If the tests of invariance show adequate fit to the model, then as an explicit second step the measurements derived from the ratings, i.e. differences in degree, can be compared. More detailed explanations of Rasch analysis are provided elsewhere. 97,99,141

It is important to distinguish these two sources of invariance in order both to understand the sources of the difference and to improve the consistency of the ratings depending on the sources of the inconsistencies.

The Multiple Sclerosis Impact Scale (MSIS-29)

The MSIS-29 has 29 items grouped into two self-report subscales aiming to measure the health impact of MS on physical (20 items) and psychological functioning (nine items). Items are summed and transformed to score from 0 (no problem) to 100 (extreme problems) for each scale. Two scores can be generated by summing items. Further details are presented in Chapter 5.

Setting, sample and procedure

We carried out a postal survey of randomly selected and geographically stratified people (n = 150) with MS recruited from the MS Society. People completed the MSIS-29 on two occasions separated by a 10-day interval.

Methods

We used two approaches to evaluate test–retest reproducibility: traditional psychometric methods and Rasch analyses. Analyses were undertaken separately for the physical and psychological subscales.

Rasch analyses

The arrangement of the data for analysis and the analyses

Two distinct data arrangements, or designs, are required to enable the analyst to answer comprehensively the questions of equivalence in kind and equivalence in degree. In one design, the item response data for the two time points are arranged horizontally (‘racked’). In the other design, the item response data are arranged vertically (‘stacked’). Each design enables both item-level and person-level analyses. The two designs provide complementary information. Neither design is sufficient on its own to answer completely the questions of equivalence in kind or degree.

Design 1: the horizontal-rating structure or ‘racked’ design

In this design, the T1 and T2 item ratings for each person are replicated horizontally (side by side). Figure 41 is a schema showing how the data for the MSIS-29 physical impact subscale were set up for analysis, and the description below applies to this subscale. An identical approach was used for the psychological impact subscale.

FIGURE 41.

Horizontal ‘racked’ data design. *Raw scores as entered. **Final patient’s worth of data entered in data set.

The consequence of this design is that the two sets of 20 item responses for each person can be concatenated (combined end-to-end) to produce a 40-item structure overall. In this way each patient can be assessed from his or her ratings on 40 items. However, because the two sets of items arise from assessments at different time points, there is an interaction between item and time. Thus, while item 1 for the T1 rating and item 1 for the T2 rating relate to the same MSIS item, the outcome on each occasion can be different due to this interaction between time and item.

With the data arranged in this manner, we then perform a Rasch analysis of the items as if they were a 40-item scale. This examines the 40 items relative to each other, and computes an item location for each of the 40 items, with associated standard errors. This, of course, gives two locations (T1, T2) and two standard errors (T1 SE, T2 SE) for each item.

Item location analysis

With the data racked, the question of equivalence of kind is addressed by determining the agreement between item locations at T1 and T2. These can be visualised by means of a scatterplot. Also, we can determine the items for which the T2 location falls outside the 95% confidence intervals of the T1 location (T1 ±  1.96 ×  T1 SE). Confidence intervals can be added to the plot to aid the visualisation.

However, here we are interested to know if the change between T1 and T2 is outside the error associated with the T1 and T2 locations. Thus, we need to compute the standard error of the difference (SED)  = √[(T1 SE)² +  (T2 SE)²], and evaluate the change (change = T1 location – T2 location) with respect to this for each item. By dividing each item’ s change by its own standard error of the difference (change/SED), we obtain the significance of that item’ s change in standard error of difference units. The significance of each item’ s change (SigChange) is interpreted as:

SigChange ≥ + 1.96 = significant improvement

0 < SigChange < + 1.95 = non-significant improvement

SigChange = 0 = no change

− 1.95 < SigChange < 0 = non-significant worsening

SigChange ≤ − 1.96 = significant worsening

We can now identify items that have undergone significant change.

Person location analysis

With data in the racked design, the question of equivalence of degree is answered by determining the agreement between predicted person locations generated by the MSIS-29 physical subscale at T1 and T2. Essentially, we determine whether the same total score at the two time points produces the same person location.

As the 40 items have been examined as a single set, the items have been located on the continuum relative to each other. That is, the 40 items have been located on a common metric. Therefore, we can examine the relationships between measurements (person locations) produced by using different subsets of the 40 items. By taking items 1–20 as one subset and items 21–40 as another subset, we can examine the extent to which the predicted person locations generated by the MSIS-29 physical subscale at T1 and T2 are equivalent. It is important to note that these are predicted locations based on the item calibrations, not the locations for the people in the study sample. Essentially, this analysis equates the two sets of values and examines the agreement between the total scores at the two time points.

Differences can also be examined at the group level using paired-sample t-tests and at the individual level by determining total scores where the T2 value is outside the 95% confidence intervals of the T1 value.

Design 2: the vertical-rating structure or ‘stacked’ design

In this design, the two separate sets of patient ratings are replicated vertically (one on top of the other) for each patient. Figure 42 is a schema showing how the data for the MSIS-29 physical impact subscale were arranged for this analysis, and the description below applies to this subscale. The same approach applies to the analysis of the psychological impact subscale. The consequence of this design is that we have twice as many people (2n) rated on the same 20-item scale.

FIGURE 42.

Vertical ‘stacked’ data design. *Raw scores as entered. **Final patient’s worth of data entered in data set = 2n.

Item location analysis

With data in the vertical design, equivalence of kind can be determined by examining if the responses to the items are stable (invariant) across each of the two occasions within the context of this design. This is achieved by examining the ICCs and the extent to which there is differential item functioning (DIF) by time point. The concept and examination of DIF is now explained in some detail.

To recap, whenever we use a school ruler or tape-measure to measure the length of objects, we have confidence that the measurement instrument is stable whatever is being measured. We cannot have that confidence with rating scales because, although we can see and handle them, the measurement rulers they subtend do not exist as visible concrete entities and measurements are inferred from people’ s responses to them. So, we need methods of determining their stability in different groups of people (e.g. men/women, older/younger, more disabled/less disabled, different types of MS, etc.). The list of different groups is of course potentially endless, so we need to be able to study the clinically important ones. Tests of item stability are called tests of DIF.

The basic premise for the stable (invariant) performance of an item is that for any level of patient disability, as best can be estimated from the data, the expected value on the item is the same irrespective of the group the person might belong to. In this study, each MSIS-29 item was rated twice by each patient: T1 and T2. Thus, patients can be classified into two groups (T1 and T2), and our aim is to determine the extent to which expected scores for each item, at the same levels of patient disability, were similar across these two ratings.

The basis for the study of DIF is the item characteristic curve (ICC), and Figure 43 shows this for MSIS-29 item 1. To recap, the ICC shows the expected or mean value for that item (y-axis) for every person location (x-axis). This is to be compared with the observed rating. Therefore, also shown in Figure 43 are the means (shown as black dots) of the ratings for four class intervals. Clearly, this should be close to the ICC curve.

FIGURE 43.

MSIS-29 physical subscale – item characteristic curve for item 1.

If the observed means follow the ICC (as they do in Figure 43), this implies that the item is functioning invariantly across the level of disability, i.e. across the trait, and that all the variation across the trait is accounted for by the estimated parameters of the model. If the observed means do not follow the curve adequately, this implies that there is residual variation not explained by the model and therefore unaccounted for. However, in addition to lack of invariance across the trait, it is possible for there to be variation across rating groups for the same level of the trait. Accordingly, the data shown in Figure 43 can be further subdivided into the relevant groups. Figure 44 shows the ICC for item 1, but with the observed means separated according to the group: T1 or T2. There is no difference. Figure 45 shows the same results for an item of the psychological subscale (item 24).

FIGURE 44.

MSIS-29 physical subscale – item characteristic curve for item 1 examining differential item functioning by time.

FIGURE 45.

MSIS-29 psychological subscale – typical example of differential item functioning over time 1 and time 2.

A unified way of assessing DIF across the rating groups, and across the levels of disability, is to consider the standardised residual of each person to each item and classify them by group (here T1 and T2) and by class interval. A two-way ANOVA of standardised residuals, in which one level of the ANOVA is the class intervals across the disability continuum and the other level gives the groups, provides a unified way of quantifying invariance of the operation of the items. In this case, the number of groups was two (T1 and T2), and the number of class intervals chosen, because of the small sample size, was four. A significant time effect, irrespective of class interval, indicates that there is a uniform DIF between T1 and T2. A significant interaction between class intervals and time indicates a non-uniform DIF. A significant difference across class intervals irrespective of time indicates a misfit to the model across the continuum. (This is analogous to the chi-squared tests of fit except that the chi-squared test of fit also enables an examination at the level of each class interval.)

Graphically, DIF across the disability variable measured by the MSIS-29 is present when the means of persons in the class intervals are not close to the expected value curve (ICC). DIF between time points is evident when, for a given estimate of the disability variable, the mean scores of the persons in the rating groups (T1 and T2) are significantly different from each other.

Multiple tests of fit inflate the type 1 error, for which the Bonferroni correction for a type 1 error level of 0.01 was applied. For the physical subscale, with 20 items, and for each item two probabilities (rater effect, rater-by-class interval interaction), we followed recommended guidelines141 and set the criterion level for misfit for each statistic as 0.05/(20 × 2) = 0.001250. For the psychological subscale, with nine items, the criterion level for misfit for each statistic was set as 0.05/(9 × 2) = 0.002778.

Person location analysis

With data in the vertical, stacked design equivalence of degree is determined by examining the agreement between T1 and T2 locations for each patient in the sample to assess if any change has occurred. A number of methods can be used. The relationship can be examined visually using a scatterplot. The degree of difference can be examined at the group level using product–moment correlations, paired sample t-tests, and ANOVAs.

More importantly, differences can also be examined at the individual person level by determining how many people in the sample have a T2 location that falls outside the 95% confidence intervals. The 95% confidence intervals around the T1 locations are computed as (T1 location ± 1.96 × T1 SE). However, as stated before, we are interested to know if the change between T1 and T2 is outside the error associated with the T1 and T2 locations. Thus, we need to compute the standard error of the difference (SED) = √[(T1 SE)² + (T2 SE)²], and evaluate the change (change = T1 location – T2 location) with respect to this for each person. By dividing each person’ s change by his or her own standard error of the difference (change/SED), the significance of that person’ s change is given in standard error of difference units. The significance of each person’ s change (SigChange) is interpreted as:

SigChange ≥ + 1.96 = significant improvement

0 < SigChange < + 1.95 = non-significant improvement

SigChange = 0 = no change

− 1.95 < SigChange < 0 = non-significant worsening

SigChange ≤ − 1.96 = significant worsening

Note that, for the MSIS-29, higher scores indicate worse disability. Thus, when change (T1 location – T2 location) is negative this implies that people have increased their scores at T2 and thus their condition has worsened.

We can now simply count the numbers of people achieving each level of significance of change and perform a chi-squared test on the results. More importantly, we can identify individuals who appear to have undergone significant improvements or deteriorations to clarify if this is the case, and if so to determine why.

Results

Rasch analyses

Design 1: the horizontal-rating structure or ‘racked’ design

Item location analyses

Table 27 (physical subscale) and Table 28 (psychological subscale) show the locations with standard errors at T1 and T2. Figures 46–49 show the scatterplots for the physical and psychological subscales with item labels (Figures 46 and 48) and with 95% confidence intervals (Figures 47 and 49). The differences between the values are small and no T2 item locations for either subscale lie outside the 95% confidence intervals of the T1 locations. These findings indicate that item locations were stable across the two time points within the context of this data design. Table 29 shows the results of the individual-level analysis of item locations. In the total sample, all items of both scales underwent no or non-significant change in their locations between the time points on each scale.

TABLE 27 - Rasch analysis: item locations and standard errors for 20-item MSIS-29 physical subscale

Item	Time 1		Time 2
	Location	Standard error	Location	Standard error
1	–0.977	0.131	–0.885	0.128
2	0.746	0.122	0.572	0.122
3	–0.110	0.118	–0.334	0.120
4	–0.304	0.122	–0.318	0.121
5	0.104	0.114	0.153	0.119
6	0.162	0.125	–0.063	0.124
7	0.035	0.119	0.167	0.114
8	–0.521	0.123	–0.373	0.119
9	0.805	0.115	1.005	0.121
10	0.664	0.113	0.680	0.111
11	0.028	0.122	–0.186	0.119
12	0.014	0.116	–0.068	0.120
13	0.078	0.121	0.077	0.125
14	0.140	0.113	0.042	0.116
15	0.511	0.119	0.559	0.118
16	–0.168	0.115	–0.233	0.119
17	0.432	0.112	0.383	0.111
18	–0.661	0.127	–0.585	0.124
19	–0.421	0.115	–0.463	0.116
20	–0.302	0.115	–0.385	0.115

TABLE 28 - Rasch analysis: item locations and standard errors for nine-item MSIS-29 psychological subscale

Item	Time 1		Time 2
	Location	Standard error	Location	Standard error
21	0.039	0.118	0.084	0.115
22	0.183	0.112	0.227	0.112
23	–0.525	0.113	–0.440	0.113
24	0.105	0.122	–0.105	0.123
25	–0.062	0.113	–0.191	0.115
26	–0.068	0.119	–0.240	0.113
27	0.125	0.122	0.119	0.123
28	0.044	0.110	0.106	0.116
29	0.333	0.116	0.266	0.115

FIGURE 46.

MSIS-29 physical subscale – plot of item locations from time 1 and time 2.

FIGURE 47.

MSIS-29 physical subscale – plot of item locations from time 1 and time 2. Black parallel lines indicate 95% confidence intervals.

FIGURE 48.

MSIS-29 physical subscale – plot of item locations from time 1 and time 2.

FIGURE 49.

MSIS-29 physical subscale – plot of item locations from time 1 and time 2. Black parallel lines indicate 95% confidence intervals.

TABLE 29 - Rasch analysis: equivalence of kind at the individual item level

Significant improvement = SED ≥ +1.96.

Non-significant improvement = 0 < SED < +1.95.

No change = SED = 0.

Non-significant worsening = –1.95 < SED < 0.

Significant worsening = SED ≤ –1.96.

Change in disability (SED)	MSIS-29 physical subscale (n = 99)	MSIS-29 psychological subscale (n = 121)
Significant improvement	0%	0%
Non-significant improvement	55.6%	50%
No change	11.1%	0%
Non-significant deterioration	33.3%	50%
Significant deterioration	0%	0%

Person location analyses

Table 30 (physical subscale) and Table 31 (psychological subscale) show the person locations, predicted from the Rasch model, implied by each and every total score on the two subscales.

TABLE 30 - Rasch analysis: person locations implied by each MSIS-29 physical subscale total score

Total score	Person location
	Time 1	Time 2
0	–5.120	–4.920
1	–4.270	–4.100
2	–3.670	–3.540
3	–3.260	–3.160
4	–2.950	–2.870
5	–2.690	–2.640
6	–2.470	–2.440
7	–2.290	–2.270
8	–2.120	–2.120
9	–1.980	–1.980
10	–1.850	–1.860
11	–1.730	–1.750
12	–1.620	–1.650
13	–1.520	–1.560
14	–1.430	–1.470
15	–1.350	–1.390
16	–1.270	–1.310
17	–1.190	–1.240
18	–1.120	–1.170
19	–1.050	–1.100
20	–0.990	–1.040
21	–0.930	–0.980
22	–0.870	–0.920
23	–0.810	–0.860
24	–0.750	–0.810
25	–0.700	–0.760
26	–0.640	–0.700
27	–0.590	–0.650
28	–0.540	–0.600
29	–0.490	–0.550
30	–0.440	–0.500
31	–0.400	–0.460
32	–0.350	–0.410
33	–0.300	–0.360
34	–0.260	–0.320
35	–0.210	–0.270
36	–0.160	–0.220
37	–0.120	–0.180
38	–0.070	–0.130
39	–0.030	–0.090
40	0.020	–0.040
41	0.060	0.010
42	0.110	0.050
43	0.160	0.100
44	0.200	0.150
45	0.250	0.190
46	0.300	0.240
47	0.340	0.290
48	0.390	0.340
49	0.440	0.390
50	0.490	0.440
51	0.540	0.490
52	0.590	0.540
53	0.640	0.590
54	0.690	0.650
55	0.750	0.700
56	0.800	0.760
57	0.860	0.820
58	0.920	0.880
59	0.980	0.940
60	1.040	1.010
61	1.110	1.080
62	1.180	1.150
63	1.250	1.220
64	1.320	1.300
65	1.400	1.380
66	1.480	1.470
67	1.570	1.560
68	1.670	1.660
69	1.770	1.760
70	1.880	1.880
71	2.010	2.000
72	2.140	2.140
73	2.290	2.300
74	2.470	2.470
75	2.670	2.670
76	2.900	2.910
77	3.190	3.200
78	3.570	3.580
79	4.140	4.130
80	4.960	4.940

TABLE 31 - Rasch analysis: person locations implied by each MSIS-29 psychological subscale total score

Total score	Person location
	Time 1	Time 2
0	–4.010	–4.250
1	–3.160	–3.380
2	–2.590	–2.770
3	–2.200	–2.340
4	–1.910	–2.010
5	–1.670	–1.740
6	–1.460	–1.520
7	–1.290	–1.320
8	–1.130	–1.150
9	–0.990	–1.000
10	–0.860	–0.870
11	–0.740	–0.740
12	–0.620	–0.620
13	–0.510	–0.510
14	–0.410	–0.400
15	–0.300	–0.290
16	–0.200	–0.190
17	–0.100	–0.090
18	0.000	0.010
19	0.110	0.110
20	0.210	0.210
21	0.310	0.310
22	0.420	0.420
23	0.530	0.520
24	0.640	0.640
25	0.760	0.750
26	0.890	0.880
27	1.030	1.010
28	1.180	1.150
29	1.340	1.310
30	1.520	1.490
31	1.730	1.690
32	1.970	1.920
33	2.270	2.210
34	2.660	2.590
35	3.240	3.150
36	4.080	3.980

Figure 50 (physical subscale) and Figure 51 (psychological subscale) plot the T1 and T2 person locations from the same total score. They are effectively superimposed.

FIGURE 50.

MSIS-29 physical subscale – plot of predicted person locations from time 1 and time 2.

FIGURE 51.

MSIS-29 physical subscale – plot of predicted person locations from time 1 and time 2.

These findings indicate that the person locations predicted by the Rasch model on the basis of the item locations at two time points are essentially equal when the total scores are the same.

Design 2: the vertical-rating structure or ‘stacked’ design

Item location analyses

Table 32 (physical subscale) and Table 33 (psychological subscale) show the results from tests of DIF over time. No items in either subscale demonstrate statistically significant differences over time (uniform DIF), or statistically significant differences in the interaction between class interval and time (non-uniform DIF). These findings imply that the items of both subscales are adequately invariant over time across the range of the disability spectrum. Figures 44 and 45, as shown earlier, are representative examples of these results from the physical and psychological subscales.

TABLE 32 - Rasch analysis: MSIS-29 physical subscale – summary of differential functioning analysis

df, degrees of freedom; F, F-value; MS, mean square.

Item	Time				Time by class interval
	MS	F	df	p	MS	F	df	p
1	0.06	0.046	1	0.831037	0.4	0.328	3	0.80547
2	0	0.002	1	0.961278	0.52	0.473	3	0.701233
3	1.33	1.214	1	0.271541	0.96	0.88	3	0.452063
4	0.6	0.515	1	0.473765	0.49	0.419	3	0.73929
5	0.24	0.38	1	0.537942	0.11	0.177	3	0.91203
6	3.6	4.094	1	0.04409	0.81	0.927	3	0.428346
7	1.93	1.6	1	0.207059	1.18	0.974	3	0.405679
8	0.09	0.075	1	0.784432	1.5	1.312	3	0.270896
9	0.06	0.042	1	0.837056	0.1	0.063	3	0.979145
10	0.49	0.371	1	0.54288	1.21	0.909	3	0.437092
11	0.67	0.914	1	0.339995	0.61	0.822	3	0.483019
12	0.01	0.011	1	0.91557	0.55	1.021	3	0.383982
13	0.24	0.214	1	0.644198	0.11	0.1	3	0.960123
14	0.38	0.335	1	0.563494	2.85	2.52	3	0.058428
15	0.12	0.163	1	0.686378	0.22	0.302	3	0.823926
16	0.19	0.194	1	0.660105	0.37	0.387	3	0.762495
17	0.01	0.013	1	0.909383	0.19	0.237	3	0.870582
18	0.01	0.013	1	0.909857	0.14	0.256	3	0.856881
19	0.05	0.047	1	0.827792	0.45	0.438	3	0.726183
20	0.32	0.179	1	0.672953	0.27	0.148	3	0.931112

TABLE 33 - Rasch analysis: MSIS-29 psychological subscale – summary of differential functioning analysis

df, degrees of freedom; F, F-value; MS, mean square.

Item	Time				Time by class interval
	MS	F	df	p	MS	F	df	p
21	0.18	0.197	1	0.657797	2.02	2.173	3	0.091665
22	0.02	0.011	1	0.918075	0.84	0.588	3	0.623498
23	1.17	1.279	1	0.259108	0.26	0.281	3	0.838798
24	0.15	0.157	1	0.691999	0.89	0.952	3	0.415992
25	1.14	1.953	1	0.163484	0.37	0.638	3	0.591391
26	0.13	0.173	1	0.677653	2.71	3.548	3	0.015102
27	0.37	0.394	1	0.530592	0.08	0.083	3	0.969362
28	0.74	0.808	1	0.369414	0.26	0.278	3	0.841192
29	0.02	0.027	1	0.869227	0.32	0.463	3	0.70839

Person location analyses

Table 34 reports the findings from the group-level analysis of person locations. Person locations generated by the MSIS-29 physical and psychological subscales at T1 and T2 were not significantly different when examined using paired-sample t-tests or ANOVAs. Standardised SRM values of 0.04 (physical subscale) and 0.07 (psychological subscale) indicated little change and therefore low potential for bias.

TABLE 34 - Rasch analysis: equivalence of degree at the sample level

ANOVA, analysis of variance; SD, standard deviation.

	MSIS-29 physical subscale		MSIS-29 psychological subscale
	Time 1	Time 2	Time 1	Time 2
Mean (SD)	0.161 (1.813)	0.186 (1.742)	–0.369 (1.517)	–0.315 (1.492)
Change [mean (SD)]	–0.252 (0.583)		–0.054 (0.722)
t–value (sig)	–0.428 (p = 0.669)		–0.739 (p = 0.462)
ANOVA F-value (sig)	0.010 (0.921)		0.063 (0.802)
Standardised response mean	0.04		0.07
Intraclass correlation	0.97		0.94

Person locations generated by the MSIS-29 physical and psychological subscales at T1 and T2 had high intraclass correlation coefficients (physical = 0.97; psychological = 0.94). Thus, both scales satisfied the recommended minimum criterion of 0.80.

Table 35 shows the results of the individual-level analysis of person locations. In the total sample, the majority of people underwent no or non-significant change between the time points on each scale (78% physical subscale; 90% psychological subscale).

TABLE 35 - Rasch analysis: equivalence of degree at the individual person level

Significant improvement = SED ≥ +1.96.

Non-significant improvement = 0 < SED < +1.95.

No change = SED = 0.

Non-significant worsening = –1.95 < SED < 0.

Significant worsening = SED ≤ –1.96.

Change in disability (SED)	MSIS-29 physical subscale (n = 99)	MSIS-29 psychological subscale (n = 121)
Significant improvement	14%	4%
Non-significant improvement	29%	50%
No change	1%	1%
Non-significant deterioration	48%	39%
Significant deterioration	8%	6%

Figure 52 (physical subscale) and Figure 53 (psychological subscale) show the relationships between the T1 and T2 person locations in this sample. The vast majority of people have a T1 location with the 95% confidence intervals of the T1 location, although there are a few notable exceptions. Note, however, that the confidence intervals for these plots are computed as T1 location ± 1.96 T1 SE, hence they will differ slightly from the determination of significance of change when the standard error of the difference is used.

FIGURE 52.

MSIS-29 physical subscale – plot of person locations at time 1 and time 2. Curved lines indicate 95% confidence intervals.

FIGURE 53.

MSIS-29 psychological subscale – plot of person locations at time 1 and time 2. Curved lines indicate 95% confidence intervals.