Assessing methods to specify the target difference for a randomised controlled trial - DELTA (Difference ELicitation in TriAls) review

Society for Clinical Trials survey

Of the 1182 members on the SCT membership email distribution list, 180 responses were received (15%). Awareness ranged from 69 (38%) for the health economic method to 162 (90%) for the pilot study method. Usage was lower than awareness and ranged from 16 (9%) for the health economic method to 133 (74%) for the pilot study method. The highest level of willingness to recommend was for the RoEB method (n = 132, 73%) and the lowest was for the health economic method (n = 28, 16%). Willingness to recommend among those who had used a particular method was substantially higher than across all respondents: the lowest level was for the opinion-seeking method (n = 40, 56%) and the highest level was for the RoEB method (n = 118, 89%).

UK- and Ireland-based triallist survey

Of the 61 surveys sent out, 34 (56%) responses were received. Awareness of methods ranged from 97% (n = 33) for the RoEB and pilot methods to only 41% (n = 14) for the distribution method. All respondents were aware of at least one of the different formal methods for determining the target difference. Usage ranged from 24% (n = 8) for both the distribution and health economic methods to 94% (n = 32) for the RoEB method. Usage was substantially less than awareness for all methods except for the pilot study, RoEB and SES methods. The highest level of willingness to recommend was for the RoEB method (76%, n = 26) followed by the SES method (65%, n = 22), with the distribution method having the lowest level of willingness to recommend (26%, n = 9). Based on the most recent trial (n = 33), all bar three groups (91%, n = 30) used a formal method. The vast majority (91%, n = 30) stated that the target difference was one that was viewed as important by a stakeholder group. Just over half (61%, n = 20) stated that the basis for determining the target difference was to achieve a realistic difference given the interventions under evaluation.

Conventional (Neyman–Pearson) approach

Randomised controlled trials allow the direct comparison of alternatives. They are overwhelmingly what can be described as Phase III (to use the pharmacological terminology) or confirmatory trials, which seek to answer the research question based on assessing ‘real’ outcomes in a substantial sample of people. Under the most common design, patients are randomly allocated to one of two treatments and statistical evidence of a difference between groups is determined (a superiority trial). Alternative designs in which the aim of the study is to demonstrate that a new treatment is equivalent or not inferior to another treatment (described in the literature as equivalence or non-inferiority trials respectively), are also possible. Irrespective of the design question, an a priori sample size calculation is required to provide reassurance that the study will provide a meaningful answer to the research question. The vast majority of trials adopt the same basic (sometimes called the Neyman–Pearson or statistical hypothesis testing) approach to calculating the sample size. 22–24 This general approach for RCTs is outlined below.

To calculate the sample size for a trial, the researcher must strike a balance between the possibility of being misled by chance and the risk of not identifying a genuine difference. Standard practice is to pick one (often the most important outcome for a key stakeholder) or a small number of primary outcomes for which the sample size is conducted. 23,25 A null hypothesis that the two treatments have equal effects is defined for a superiority trial. It is this null hypothesis that the trial is attempting to assess evidence against. Rejecting the null hypothesis when it is true (type I error) would lead to a concluding that one treatment is superior to another when in reality there is no real difference in treatment effect. The significance level of the test (α) is the probability of the occurrence of a type I error, that is, falsely concluding that there is a difference when there is not. Failing to reject the null hypothesis when it is false (type II error) would lead to a trial concluding that one treatment is not superior to another when in reality one treatment is superior. The probability of the occurrence of a such an error is β, which is equal to 1 minus the (statistical) power of the test. The smaller the specified difference to be detected the lower the power for a given sample size and significance level.

Once these two criteria are set, and the statistical tests to be conducted during the analysis stage are chosen, the sample size is determined depending on the magnitude of difference to be detected. This ‘target’ difference is the magnitude of difference that the RCT is designed to reliably investigate. It is worth noting that in practice the value used as a target difference might be one that is considered important or arrived at from another basis, such as what would be a realistic difference26 as observed in previous studies or a difference that leads to an achievable sample size. Different methods can be used to determine this target difference (Box 1; see also Chapter 2 for details). This issue is considered in more depth in Chapter 4 (see Specifying the target difference).

For an equivalence (or non-inferiority) trial, as opposed to a superiority trial, a range of values around zero will be required within which the interventions are deemed to be effectively equivalent (or not inferior) in order to establish the magnitude of difference that the RCT is designed to investigate. The limits of this range are points at which the differences between treatments are believed to become important and one of the treatments is considered superior; the difference between one of these points and zero can be viewed as defining a minimum difference between treatments that would be important and which the study is designed to reliably investigate. The sample size calculation can also incorporate the possibility of an expected ‘real’, although non-important, difference into the calculation, reflecting that two treatments are very unlikely to have a (statistically) identical effect. A hybrid approach exists between the superiority and the equivalence/non-inferiority designs, sometimes called ‘as good as or better’. Under this approach a ‘closed’ testing (i.e. ordered) procedure is used that allows both an inferiority and a superiority question to be potentially answered in a single study. 23 As with the standard superiority and equivalence/non-inferiority designs, a definition of the difference to be detected is still needed.

Once the target difference, or for an equivalence/non-inferiority trial the limit(s) of equivalence, is determined then the method of estimating the sample size will depend on the proposed statistical analysis, trial design (e.g. cluster or individually randomised trial) and statistical properties specified (e.g. agreement for paired data). The general approach is similar across studies under the Neyman–Pearson approach. Implicitly, the study is designed as a ‘stand-alone’ evaluation (without the resort to any external data) to answer the research question.

Other statistical approaches to defining the required sample size are Fisherian, Bayesian and decision-theoretic Bayesian approaches, along with a hybrid of both the Bayesian and the Neyman–Pearson approaches. 22 Economic-based methods tend to follow a decision-theoretic approach, with the types of benefits, harms and costs included reflecting the perspective of the research funder and the values used and the way that they are combined (typically a net benefit approach) reflecting the context of the study. Despite the existence of these different methods a recent review of RCT sample size calculations identified only the Neyman–Pearson approach in widespread usage. 22 Regardless of the statistical method used the key issue is what magnitude of a difference is of practical interest and one that the study should be designed to detect. As will be described in The relevance of decision theory-based models, the intent of the economic-based approaches can be different as they tend to focus on maximising some measure of efficiency. Prior evidence may be informally guide the process though explicit incorporation of prior evidence in the sample size calculation for a RCT is also possible. 27,28

Bayesian approaches to the design and analysis of randomised controlled trials

Much has been written about the benefits of Bayesian approaches to the design and analysis of RCTs. 24 They allow incorporation of current evidence (represented as a prior distribution) with new evidence to produce a summary of the overall evidence (posterior distribution). With substantive computing power now readily available, their use has grown because of the flexibility of the approach and intuitive appeal. One area in which the value of using Bayesian methods has been highlighted is for sample size determination of RCTs. Standard sample size calculations assume that the imputed values are ‘known’ (i.e. without any uncertainty); in reality, inputs such as event proportions or variances are estimates of what is anticipated to happen in the trial. A Bayesian framework provides a natural framework in which uncertainty about the inputs can be incorporated (as a prior distribution) and the impact of uncertainty on the estimates can be included to evaluate a predictive distribution of power. 24 Following such an approach still requires specification of the difference to be detected. The use of Bayesian methods for design and analyses does not avoid the need for interpretation of the importance of the results. Although they allow a more comprehensive, although also complex, representation of evidence through posterior distributions, consideration of the relevance of the findings is still necessary. To state that the outcome (posterior) distribution differs between treatments, or that the distribution of the mean difference excludes zero, does not clarify whether we should act on the basis of the result. Correspondingly, ‘Bayesian p-values’ (e.g. the posterior predictive probability that the outcome is better for treatment A than for treatment B) are similar to frequentist p-values in that for the same reasons (as noted above) they are an insufficient basis on which to make a decision. One approach to this issue is to define an ‘indifference zone’ in the outcome distribution in a similar manner to that adopted in equivalence trials. 29 The CHART trial30 provides an example of using clinicians’ opinion (both a range of equivalence and expectations regarding a realistic difference) regarding 2-year survival for trial design and data monitoring; this approach could be readily used a priori to determine the corresponding sample size within a Bayesian framework. A Bayesian approach can naturally be extended into a formal decision model, which enables consequences of the decision to be formally incorporated into the analysis.

Adaptive designs

Adaptive designs are studies that are set up to formally modify the trial design, according to accumulated data. They are used most commonly within the pharmaceutical setting for Phase II studies in which the primary outcome can be assessed in the short term after randomisation. A decision about adapting the trial design can therefore be made while the study is still recruiting. Such adaptive designs can be viewed as a form of decision model. Typically, they need specification of what is viewed as important (e.g. outcome values that would lead to the intervention not being worthwhile to pursue) at the outset to control the process of adaptation. Common types of adaptation are discarding a treatment arm as it is unlikely to achieve the desired or optimal effect (e.g. Phase II dose-finding study)31 or stopping a trial if benefit/futility has been shown (sometimes called group sequential trial). 29,32 Although the latter may have a decision rule that involves only a statistical rule of precision (e.g. p-value or Bayesian posterior probability), decision-making of Data Monitoring Committees and study investigators will involve more than the pure statistical results of accumulated data, with consideration of the consequences of making a decision such as stopping early. Some Phase II designs adapt based on a maximum allowable sample size and/or decision rules based on Bayesian posterior probability. 29 Trend analysis models have been proposed that adjust the sample size according to the trend in the current data. 33 They are not commonly used because of the risk of bias (similar to play-the-winner designs)34 with regards to inflated type I and type II errors and because knowledge of the adaptation reveals the direction, and possibly magnitude, of the observed difference at that time point. In contrast, Phase III trials will typically be designed to detect or exclude a particular (target) difference.

The relevance of decision theory-based models

As highlighted earlier in Why seek an important difference?, the decision that the study is designed to inform, such as treating patients with lower back pain with surgery or conservative treatment, should guide the design of the study. Decision theory-based models allow the decision and corresponding losses (consequences and costs) to be quantified and incorporated in an extended model that incorporates the clinical evidence and any other relevant data. 35 The underlying rationale is that any decision should be made to maximise utility (value) given current evidence. There has been much discussion about whether RCT results should be analysed on their own or whether a decision modelling approach should be adopted that incorporates the decision (and corresponding consequences) which the study is seeking to inform. 36 For example, should the spinal surgical treatment (from the Norwegian Spine Study) be analysed on its own or should prior evidence be incorporated into the model, all with estimation of the impact of changing the treatment. It has been noted that investigators who design and carry out a study are generally not the same as those who make decisions. 24 As such, there is a practical challenge in implementing such an approach. However, models have been proposed that specifically address the design of RCTs: they seek to determine the optimal sample size given current evidence and the estimated consequences of the actions. 37 Although clearly of a different nature to other approaches, such as the MICID and variants, these models may be viewed as informing the same decision (but taking into account other factors); as such, they are also considered in this monograph.

Brief description of the anchor method

In its most basic form, the anchor method evaluates the minimum (clinically) important change in score for a particular instrument. This is established by calculating the mean change score (post minus pre) for that instrument among a group of patients for whom it is indicated (using an additional measurement – the ‘anchor’) that a minimum clinically important change (or difference) has occurred. The word ‘minimum’ reflects the separation of patients according to the amount of change, with the smallest amount of change viewed as ‘important’ calculated. In evaluating a patient’s progress (e.g. before and after a given treatment), a clinician could use this value as an indication of the expected change in score of an instrument among patients who have indicated on the anchor measurement that they consider themselves (or are considered by someone who completes the anchor measurement on their behalf) to have undergone an improvement within a particular time frame.

Variations on the anchor method

An anchor method example is given in Box 2. However, various implementation approaches of the anchor method have been used (i.e. varying the number of points and labels attached) along with variations in how an anchor is used to determine an important difference [typically the M(C)ID]. One of the most common variations is not only to consider the mean change score for improved patients but also to take account of the relative change compared with another subset of those undergoing assessment (who have been tested using both the same instrument and the anchor) who reported no change over time. 54–58 The results for both groups can be compared, potentially allowing the calibration of a resulting MCID value by adjusting for the mean value found for the ‘no change’ group (i.e. variation in score that is unimportant). Yalcin and colleagues59 refer to the original method as ‘within-patient’ change and this variation as ‘between-patient’ change. However, these terms are not solely used for these variations and can therefore be confusing. The variations are depicted in Figure 3. Another very common variation is to consider the percentage change score in the instrument under consideration,60 rather than simply using the absolute change score. Including the percentage change score as well as the absolute score has been justified as possibly reducing variation, thereby ensuring that the MCID values obtained are more robust. Other variations reported used the median change rather than the mean change since baseline. 61–63

FIGURE 3.

Anchor method illustrative example.

Diagnostic accuracy methodology has also been used to define the MCID with an anchor method. The anchor response is used as the reference standard, and the sensitivity and specificity of a cut-point difference in the instrument under consideration is assessed against the anchor definition. Use of the cutoff that optimised together sensitivity (proportion of those who had experienced an important difference correctly identified) and specificity (proportion of those who did not have an important difference correctly identified) was almost universal. In a few cases this was not carried out, for example using a cut-off to maximise sensitivity over specificity, unless the point nearest the left-hand corner of the sensitivity versus (1-specificity) plot gave a particularly low value for sensitivity,64 or using an 80% specificity rule. 65,66 In other studies, a receiver operating characteristic (ROC) curve analysis was undertaken but was not used to calculate the MCID. 67,68 In other instances, diagnostic accuracy data were reported but a ROC curve approach was not used to determine the estimate. 69 When multiple MCID estimates were generated (e.g. from different anchor definitions or by using different anchors), an average or triangulation of the results was sometimes used. 70–72 Other less commonly used methods to derive a MCID value from an anchor-based method included various types of regression models [e.g. analysis of variance (ANOVA), logistic regression, Rasch analysis, classification and regression trees (CART) analysis, mixed-effects models and linear discriminant analysis]. 73,77 Harman and colleagues78 developed a logistic regression model for the probability of experiencing a negative life event based on prior mental health levels. A substantially different approach was proposed by Redelmeier and colleagues79–83 in which other patients were used as anchors on which a patient could rate his or her own health (or health improvement) in comparison to others.

Aside from variation in the method used to elicit a MCID value, there was variation in the anchor itself and in the ‘transition question’ used to determine whether patients had improved or not. A 15-point Likert scale was often used as the anchor,84–86 based on the method established by Jaeschke and colleagues,87 although five-point69,88,89 and seven-point54,90–92 scales were also widely used. A visual analogue scale (VAS) can be used (e.g. with < 10 mm considered ‘unchanged’ and > 30 mm considered more extreme than minimal change). 93 Existing instrument questions were also adapted, including the Short Form questionnaire-36 items (SF-36) global change score, Disabilities of the Arm, Shoulder and Hand (DASH) and the (modified) Rankin score. 77,94–97 For example, Khanna and colleagues95 used multiple anchors, including the SF-36 global change score and four additional items from the SF-36 relating to walking and climbing stairs. Several studies utilised an item from the instrument that was being assessed for MCID values as the anchor question for other items within the same instrument. 74,98–100 For example, Colwell and colleagues74 used the pain frequency and pain severity items within the Gout Assessment Questionnaire (GAQ) as anchors by which to gauge the MCID values for other items within this outcome.

In many cases, the original anchor categories were later merged because of sample constraints (e.g. poor literacy among respondents or an insufficient number of patients in a particular category of improvement to enable analysis to be undertaken separately for that category). 101,102 In some cases, if the anchor considered deteriorating states as well as improvement, the sign on the deterioration point of the Likert scale was changed and its participants were merged with the improving patients to give a larger number of patients in the anchor groups (although this makes the strong assumption that MCID values will be identical for improvement and deterioration). 103 For ROC curve analysis to be used within the anchor method, the anchor question must be dichotomous (those experiencing a MCID vs. those who did not experience a MCID). There is therefore a need to split the anchor in some way. This often requires an arbitrary decision about what points on the Likert scale constitute important change in comparison with no real change. Consensus between two colleagues was used in one instance to derive a definition for important change, along with the Rasch method to find ‘natural’ cut-points for the data. 75

The anchor question was most often posed to patients alone63,104 although in some cases the clinicians’ views alone were used, even in instances when the instrument under consideration was a patient-reported outcome. The opinion of a clinician was also used instead of a patient-reported outcome,105 or in conjunction with a patient’s score, either to determine the extent of agreement between patient and clinician scores or to average the score (e.g. to reduce potential patient biases) by incorporating both patient and clinician views. 69,71 Parents’ views were also occasionally used, especially for studies of paediatric outcomes. 106–109 Anchors do not necessarily have to be based on individuals’ opinions; ‘objective’ measures of improvement can be used (e.g. > 5 mm healthy toenail growth). 110 Other studies used retrospective analysis, for example the mean change score among those who were later hospitalised compared with those who were not later hospitalised. Readmission and/or death was also considered in this way78,111,112 If a treatment of known efficacy was the intervention that patients received, the mean change score was used as patients were expected to show clinically important improvements over time. 113 It is worth noting that the anchor method was not always successful in deriving values for interpreting an important difference. 114,115

Summary of anchor studies

A total of 447 studies were found that used the anchor method; 194 of these also used another method (see Combination of methods for details). Several studies had split their sample into separate cohorts (e.g. if the data came from two separate clinical trials). 76,116 In most instances, a rationale for calculating the MCID was not explicitly given. As many studies looked at instrument development, calculation of the MCID often formed part of a wider array of reliability and validity testing of new instruments; other studies merely noted that for a particular instrument (or use of an established instrument for a particular disease/condition) the MCID had yet to be established. Future sample size calculation was very rarely cited in the rationale for calculating the MCID. 116

Across all its different possible forms, the anchor method was used in a wide range of specialties including, but not limited to, surgery,117 orthopaedics,70 paediatrics,109 rheumatology,69 stroke rehabilitation,118 cancer,119 urology,120 respiratory medicine,75 mental health67 and emergency medicine. 121 Almost all cases looked at quality-of-life measures, particularly with regard to pain,85 function and mobility. 122,123 The method is not used exclusively for establishing the MCID for new or existing quality-of-life tools; non-scale-based outcomes were also considered. These usually related to patient physical functioning (in many cases in conjunction with a wide range of instruments for a particular condition being assessed at the same time), for example a 5-minute walk test, 1-minute stair climbing,94 comfortable gait speed77 and (more unusually) the number of palms of your hand it would take to cover up all of the psoriasis on your body. 61 The method can be used for acute124 or chronic125 conditions.

Practical considerations for use of the anchor method

A number of common issues with regard to applying the anchor method were highlighted. In particular, the generalisability of a determined MCID value was often queried. In some instances this was simply in relation to the size of the sample,75,94,105,118,121,122 which in itself is not an problem exclusive to studies determining important difference values. However, in other instances the range of applicability of the resulting MCID value may be compromised. 49,94,118,121,126 For example, a MCID value for a chronic progressive illness may not be applicable to a newly diagnosed population if the duration of illness among most participants surveyed to determine the MCID value is far longer. 116,127–130 Although it is, in principle, possible to account for this in any future anchor method studies (e.g. by providing a breakdown of MCID values for subgroups based on duration of illness), in practice there may be more complex confounding issues. Long-term sufferers of a chronic condition may have different expectations of change than a newly diagnosed population. In addition, severity of illness may not necessarily be directly correlated with duration, and this may be a more important aspect to consider. In fact, the effect of severity of illness on identified MCID values was often examined by study authors; however, not all studies found a statistically significant effect. 63,71,105,127,131

Several studies that had considered separately MCID estimates for improvement and deterioration noted differences in the values derived for these groups. 75,88,95,121,126 Although this, as with other generalisability concerns, may be related to participants’ expectations or regression to the mean, it is important to consider whether an assumption of linearity, with no change centred on zero difference, is appropriate. The decision to merge both improvement and deterioration groups (e.g. because of a small sample size) to derive one singular ‘change’ value instead may be inappropriate if the size of an important change is different for improving participants and deteriorating participants. 49,54,95,122

The validity of the anchor used was also cited as being potentially problematic from a methodological point of view. Likert scales were most commonly used as anchors to derive the MCID value, but there was considerable variation in the number of point options provided within the Likert scale. Differing sizes of anchor scales may be appropriate as the change experienced by participants, and their ability to differentiate, will vary between contexts and study populations. In addition, there may be other factors that influence the size of the Likert scale used. For example, one study noted that, if the population of interest has poor literacy, a Likert scale with a large number of options may be less suitable than one with fewer points for respondents to consider. 132 Several studies using Likert scales with a large number of options later had to merge multiple options together because so few participants had selected each option, thereby retrospectively creating a smaller Likert scale that might have been more suitable in the first place.

A more fundamental consideration when using the anchor method is how to decide on an appropriate cut-off point to apply to the anchor responses. Although any formal approach for determining this cut-off was seldom mentioned as part of the study methods, several study authors pointed out the difficulty of choosing a particular value. 59,80,88,110,116 For anchor studies using a ROC curve approach, choosing a cut-off point on the Likert scale to dichotomise the sample into those who did and those who did not experience change is essential, but the choice of cut-off is arbitrary and may compromise the extent to which an anchor method detects or fails to detect important change. In some instances, multiple cut-offs are analysed, or mean change values are reported for every different point on the Likert scale. This is useful as it allows those wishing to utilise MCID values to use the value most suitable for their own needs. Conversely, it could be argued that, by not choosing an appropriate cut-off and instead providing every possible value of clinical importance from the anchor responses, studies fail to provide guidance for which MCID values are more likely to be realistic overall. Some papers focused on only one choice of definition; this was particularly the case for papers that had used a ROC approach. Reporting of anchor methodology was sometimes insufficient to allow reanalysis of the data if an alternative anchor definition of important change is desired.

More generally, the validity of both the outcome under consideration and the anchor tool being used to define its MCID were noted as important. 85,88,122,126,133 Validity issues apply more widely to the development of any new outcome for health measurement. For example, given an ambulatory population, the outcomes used must be able to detect small changes and be free from ceiling effects, whereas for a population with severe disease, the possibility of foor effects could be more relevant. 69,127 Information on a outcome’s ability to detect change in the first place complements MCID values and helps determine whether or not a point estimate of change is suitably relevant. 63,110 The inability to achieve a high level of discrimination (e.g. good diagnostic performance if a ROC curve approach is used) would suggest that a MCID for a particular outcome cannot be reliably determined using this anchor. Possible reasons for poor performance include a disconnection between the anchor and the outcome (e.g. difference in perspective) as well as the properties of the outcome itself (e.g. low reliability).

With regard to the population of interest, the generalisability of the study population is not the only aspect of the study conduct that requires methodological consideration. One commonly cited difficulty in using the anchor method relates to the potential for participants to exhibit recall bias over the course of the study period. 70,75,85,126,129 Although this concern is not exclusive to the calculation of a MCID, it is relevant to consider the possible impact on the study estimate, particularly in instances in which there may be additional concerns about study methodology (e.g. variation in length of participant follow-up). 76,116,129,132 There are examples of methods used to counteract this bias. For example, Wyrwich and colleagues,134–137 at each data collection session, asked participants what they were doing that day. At the following session, to aid their memory, their answer from the previous session was relayed to them as a reference point to help them recall how they had felt at the previous point in time. Response shift may also be problematic, as participants’ perceptions of what amount of improvement they would be satisfied with may change during the course of their treatment, and their expectations may not be consistent over time. 138

The method proposed by Redelmeier and colleagues,80 in which other participants act as the reference point avoids recall bias because all data can be collected at the same time. However, the data analysis is more complex83 and this approach is not universally appropriate for calculating the MCID in all contexts. For example, participants might find it difficult to discuss particularly sensitive or private health issues with others. A more general consideration with regard to patient involvement is the value of their MCID scores in comparison to (most commonly) clinicians’ views of the extent to which important change had occurred. The development of methods to elicit an important difference has occurred alongside increasing value being placed on the experience of patients and their increasing autonomy as experts of their own illness. 70,120,128 However, in some circumstances (e.g. paediatrics) it may be appropriate to consider the views of others in addition to (or even superseding) the views of patients. It has also been noted that clinicians’ views of whether or not important change has occurred may not be independent of patients’ own views on this. 69 Patients’ own views on their progress may be shaped by the views of their clinician. It is therefore important to note the methods used in data collection with regard to whether or not clinicians and/or patients were blinded to the views of others when responding to the question of whether or not the patient had experienced important change. This would establish whether or not the values elicited are potentially biased. Consideration of data collection methods is also important with regard to minimising non-completion. Using interviews rather than survey questionnaires for this purpose may, in addition to improving completeness of data collection,68 also minimise the potential for misunderstanding the meaning of a question being used. 118 However, this is a resource-intensive approach leading to a small number of participants likely being involved.

Common psychological biases may be an issue (e.g. a ‘gratitude factor’ or halo bias) where anchor responses are more favourable than is realistic. 63,122 Potential patient biases may also depend on the effect of a particular outcome on a patient’s own circumstances. For example, in a health-care system in which financial assistance requires proof of continuing ill health, a patient may be reluctant to express that an important improvement has occurred. 63,105 It is also widely acknowledged that an overall average MCID estimate may not correctly quantify whether or not important change has taken place for an individual patient, as there will of course be variation between patients. 76,85,95,122,132,133 Furthermore, the MCID estimate may vary between studies. The agreement or otherwise of the estimate with different studies should be explored. 54,64,85,121,126

Triallists wishing to use a MCID estimate derived using an anchor method should review the methodological quality of the anchor method study, specifically with regard to the above mentioned issues, to establish whether or not the MCID value is valid for the future trial population of interest. Researchers wishing to use an anchor method to establish the MCID of a particular outcome should consider these issues so that they can report variation and potential biases as explicitly as possible.

Brief description of the distribution method

The distribution method typically determines a value that is larger than the inherent imprecision in the measurement (e.g. MDC) and which is therefore likely to represent a meaningful difference or what can be statistically detected (e.g. minimal statistically detectable difference). Other approaches are based on the nature of the outcome (e.g. a fraction of the response range for a VAS). 139

Variations of the distribution method

Summary of distribution studies

A total of 324 studies were found that used the distribution method; 153 of these also used another method (see Combination of methods for details). The majority of the papers reported results from studies in a range of clinical areas, including back pain, diabetes, mobility, quality of life. Several papers reported on non-clinical areas, mainly toxicity testing. The outcome of interest varied and included biochemical markers, degree of flexion, depression, oxygen output, quality of life, time, radioactivity level and whole effluent toxicity.

Practical considerations for use of the distribution method

Of the three basic types of distribution method identified – the measurement error-based (including RCI approach), statistical testing-based and rule of thumb approaches – the first two are widely used in important difference assessments, but neither translates cleanly into the target difference context. Statistical testing-based approaches clearly cannot be used for specifying the target difference in a sample size calculation; in an important difference setting the assessment is carried out after the data are collected and therefore the sample size is a known and fixed quantity, whereas in a RCT sample size calculation context the sample size is the object to be determined, which in turn depends on the target difference. The measurement error approach also does not translate straightforwardly; such an assessment is typically based on test–retest (within-person) data, whereas many trials are of a parallel group design (i.e. assessing a between-group difference). It may also be difficult to obtain consistent and reproducible measurements for some patient populations and/or conditions. 212 The rule of thumb approach is dependent on the outcome having inherent value. It is akin to the approach often adopted for defining an equivalence/non-inferiority margin in which a substantial fraction (e.g. a third or a half) of the previously observed effect is used. 23,213

Brief description of the health economic method

The approaches included under this method make use of the principles of (health) economic evaluation and typically involve defining a threshold value for the cost of a unit of health effect that a decision-maker is willing to pay and using data on the differences in costs, effects and harms to make an estimate of relative efficiency. In some respects this might be thought of as an explicit analytical expansion on the sentiment expressed in the MCID.

Variations on the health economic method

The two earliest papers identified214,215 used simple manipulation of costs and effects data to identify the difference in effectiveness that would lead to the incremental cost per unit of health effect being no more than a given threshold that represents the decision-makers maximum willingness to pay for that health effect. As an example of the approach the steps involved in Detsky’s214 approach are outlined in Box 3. With Detsky’s approach there is an implicit assumption that the target differences compared in the cost-effectiveness analysis are meaningful. Furthermore, although the method attempts to weigh up the costs and benefits of conducting the research, the spectrum of costs considered is very narrow. For example, the consequences for costs following a change in treatment are not considered. Most of the other approaches can be viewed as an elaboration of this general approach.

An alternative approach is typified by Torgerson and colleagues. 215 They likewise considered that the relevant issue to be the economic ‘significance’ (or importance) of a difference. Their definition of significance was based on the difference in effectiveness that would, for a given difference in cost, ensure that the incremental cost per unit of effectiveness was no greater than some prespecified threshold value or that the average cost per unit of effectiveness was equivalent between the two procedures. The sample size was determined using the conventional approach although the difference deemed important was based on the estimated cost differential and control group outcome. The key distinction between the approach of Detsky and that of Torgerson and colleagues is that, in the former, the difference in effectiveness between interventions is compared with the cost of the research whereas, in the latter, the difference in effectiveness between interventions is compared with the difference in costs of treatments between interventions. In both, only rudimentary consideration was given to what costs and effects should be, and neither considered the imprecision in cost estimates.

In Table 3 two simple scenarios are illustrated using data from the National Institute for Health and Care Excellence (NICE) appraisal comparing laparoscopic surgery with open surgery for inguinal hernia repair. 216 In the first worked example, the costs of using both interventions have been estimated and these data are used to estimate the target difference in effectiveness (measured in terms of QALYs) if an incremental cost per QALY of £20,000 is used (a threshold value adopted by NICE in England). In the second example, which compares average (or, more correctly, marginal) costs per QALY, the target difference in effectiveness is estimated from information on the costs of the two intervention and information on the QALYs provided by open repair (the control intervention).

Following the same general principle as Torgerson and colleagues,215 Samsa and Matchar217 used an economic model to estimate the difference in effectiveness of a treatment for stroke that would be needed so that the costs of treatment would be offset by cost savings from disabilities avoided. The authors’ economic decision rule was that the introduction of the intervention should be at least cost neutral, but a cost-effectiveness threshold could have been readily used (and would have resulted in a smaller MCID being identified). Briggs and Gray17 took an approach that in its basic form was conceptually similar to that of Torgerson and colleagues. 215 It assumes that a maximum value for incremental cost-effectiveness is defined. From this, a value for the difference in cost can be calculated such that the critical difference in cost (δC) is equal to the threshold value for the decision-maker’s willingness to pay for a unit of health effect (R_c) multiplied by a predefined difference in effectiveness ( ΔẼ) . More formally this can be represented as:

It is unclear how ΔẼ, the predefined difference in effectiveness, should be determined. ΔẼ might be defined as the MCID and one of the other methods (e.g. anchor) used to provide an estimate. When comparing two interventions the hypothesised difference in cost (ΔC) should be less than the critical difference in cost (δC). If the hypothesised difference in cost is greater than the critical difference in costs then the trial should not be performed.

As a response to the approach outlined by Briggs and Gray,17 O’Hagan and Stevens19 proposed a Bayesian method of assessing sample size. Central to their approach and in common with the other approaches is the identification of a threshold value for the decision-maker’s willingness to pay for a unit of effectiveness. For any given threshold value the objective is to maximise the net monetary benefit (NMB), where the NMB is the product of the difference in effects between two interventions multiplied by the threshold value and minus costs. More formally it can be represented as:

NMB = R_cΔE-ΔC

where ΔE and ΔC are the differences in effectiveness and costs, respectively, between interventions and R_c is the decision-maker’s willingness to pay for a unit of effectiveness. The critical issue is that a decision in favour of one intervention would be taken if the NMB is > 0. The Bayesian approach requires that prior distributions are specified for the set of unknown parameters used to derive the NMB. These parameters (in the simplest form) might be costs and effects for the interventions compared or they might be the parameters that are the determinants of costs and effects. The approach does not require that any comparison is made between the parameter estimates used to calculate the NMB for the interventions under investigation. However, this implies that any difference in values for a parameter between interventions is potentially important (as the decision rule is such that any value other than 0 for NMB informs the choice between interventions). It would be possible to include prior information about what is a meaningful difference between interventions for a given parameter. This would require the use of another (e.g. opinion-seeking) method.

Other analysts have considered the identification of optimal sample size from the perspective of a profit-maximising frm. 218,219 These analysts define an expected net gain, that is, profit function, the objective of the approaches being to maximise the expected net gain from research. The implied minimum value of this function is 0. Both of the approaches outlined use a Bayesian decision-theoretic approach to sample size determination. How judgements are formed about differences in effectiveness, costs or cost-effectiveness (from the perspective of the payer of health care) is not specified. Gittins and Pezeshk218 suggest that the extent to which a therapy is adopted corresponded to a personal threshold for the difference between interventions. How this is arrived at is unclear.

Kikuchki and colleagues220 and Willan219 describe an extension of the approach outlined above by considering the optimal sample size from the perspective of some single payer system. In both approaches, the objective is to maximise a net benefit function. The assumption with the net benefit approach is that any positive value is important. The approach incorporates the cost of research, the likelihood of adopting a more beneficial treatment and subsequent effects on costs and effects of changing management. Again, however, how judgements are made about whether differences in the input parameters are important is not outlined.

Willan and Eckermann38 provide an example of the use of modelling to determine the expected value of sampling information and the expected trial cost. The optimal position for the decision-maker is when the difference between the expected value of sampling information and the expected trial cost is maximised. The expected value of sampling information in simple terms describes the monetary value of removing uncertainty surrounding a decision to adopt an intervention. It is based on the net benefit statistic described above and as such requires the definition of a threshold value for a decision-maker’s willingness to pay for a unit of a health effect and an assumption that any positive value of net benefit is important. The results of the analysis are dependent on the starting values used in the model and the structure of the model (both of which make implicit or explicit assumptions) and important differences between interventions. For example, excluding an event from the model assumes that there is no meaningful difference between interventions. This assumption may be explicitly based on other information about what constitutes a MCID or may implicitly assume that there is no magnitude of a difference that is important. Furthermore, as with O’Hagan and Steven’s19 approach, any difference in values for a parameter between interventions is potentially important (unless some further decision rule on what constitutes a MCID is explicitly built into the analysis). In contrast to the above studies, Bacchetti and colleagues221 argue that looking at the value of information is difficult and risky for investigators planning a trial because the complexity of the method and the number of assumptions required make it easy for peer reviewers to raise concerns. They suggest that when deciding on the sample size of a trial the ‘cost-efficiency’ should be maximised, where cost-efficiency is defined as the ratio of the expected scientific/clinical/practical value (v) of the study for a given sample size (n) to the cost (c) of conducting the study for a given sample size (n), that is, cost-efficiency = v_n/c_n. Given the difficulties in establishing value, which forms the basis of their criticism of the value of information approaches, they argue that it is possible to specify general properties of how sample size influences projected value and that this relationship, along with the cost of research, can be used to identify the sample size of the study. The decision rule is that cost-efficiency should be maximised and that any increase in cost-efficiency is potentially important. However, the definition of value used is uncertain.

Summary of health economic studies

Thirteen studies were identified that used a health economic method; none used another method as well. These studies are illustrative of a generally progressive development of methodology over time, with approaches becoming increasingly sophisticated. All have used some degree of modelling and all based conclusions on an economic decision rule; as such, the use of terminology such as ‘minimally important difference’ was infrequent and generally absent in the most recent studies, which is unsurprising given that later papers were based on explicit critiques of traditional methods for sample size determination, which require a target difference to be defined. All of the methods described made use of hypothetical examples and, based on discussions with experts in this area, the use of economic methods in practice has been very rare.

Practical considerations for use of the health economic method

The economic methods in this section describe increasingly complex approaches to weigh up the difference between interventions over multiple outcomes (different measures of effectiveness, harms, uptake, adherence, costs of interventions, costs of research, etc.). Which outcomes are included will be determined by the decision problem faced. The methods describe an increasingly comprehensive consideration of the outcomes and an increasingly sophisticated way of modelling their joint impact. Nonetheless, to do this they have to define a decision rule, for example, treatment x is better than treatment y when the NMB for x compared with y is > 0. The perspective of the decision-maker is clearly critical in what should be considered. It is unlikely that one analysis would satisfy all relevant perspectives – those of the patient, clinician, funder and health-care policy-maker. However, it may be easier to see such an approach being adopted by a research funder to provide reassurance that a particular study is worthy of the research cost incurred. The more sophisticated modelling approaches also are required to make implicit or explicit assumptions about the value of a target difference between treatments for the individual parameters that are used to estimate NMB. How this is carried out is not addressed by the health economic approaches. Such methods can be viewed as a comprehensive framework for decision-making that incorporates judgments about what would be clinically important and relevant empirical evidence.

A major issue with the use of such methods, as with any economic evaluation, concerns the data sources used to supply the required parameter inputs. This includes an understanding of the relationship between parameter input values, for which little robust data are often available. The expected value of sampling information approach outlined by Willan and Eckermann38 assumes that the current evidence is sufficiently summarised to be able to evaluate the incremental benefit of a future study in reducing uncertainty. This implies an up-to-date economic evaluation, which in turn implies the systematic assembly of current clinical and other evidence. Although the methods have intuitive appeal, the sophistication of the approach along with the underlying data demands perhaps explain their limited use to date. The more comprehensive the model structure the more challenging it is to populate the model. Assumptions for key aspects of the model, such as the time horizon, which defines the period over which the model will incorporate the costs and consequences of the intervention decision, may be difficult to predict and would therefore require sensitivity analyses to be conducted. Narrowing the scope with regard to which costs and benefits need to be considered, for example by adopting the perspective of a profit-maximising company, would simplify the approach.

Basing the target difference on a clinical outcome, as per the conventional approach, furthers the science of interventions by producing a result that directly assesses the intervention’s clinical outcome in isolation from other factors (e.g. costs). This also has the intuitive appeal of simplicity of interpretation for the patient, health-care professional and funder. Given this, it seems unlikely that the health economic method would currently be accepted as the sole basis for specifying the target difference. It should be noted that determining the study size (target difference) under the conventional approach alone, as is the typical current practice, may lead to a study that cannot reliably answer the corresponding health economic question. 215

Brief description of the opinion-seeking method

The opinion-seeking method determines a value (or plausible range of values) for the target difference by asking ‘expert(s)’ to give their view on what value or values would be reasonable. This requires participants to be presented with information (either real data or hypothetical scenarios) in order for them to provide their opinions. Methods for eliciting opinions and for establishing a consensus to consolidate these opinions of relevant groups of individuals vary. A target difference that is realistic, important or both could be sought.

Variations on the opinion-seeking method

There can be wide variation regarding the most relevant group of people to seek an opinion from (e.g. patients, clinicians, triallists); the method of selecting individual experts (e.g. literature search, mailing list, conference attendance); and the number of experts consulted. Other variations include the method used to elicit values (e.g. interview, survey), the complexity of the data elicited and the method used to consolidate results into an overall value or range of values for the difference.

Most studies solicited the opinion of clinicians (across specialties and/or settings and/or disciplines), although others asked the opinions of patients,124,222–225 the views of both clinicians and patients,135,226–229 or the views of multidisciplinary experts (e.g. including non-clinicians such as triallists). 41,230–236 The methods used to recruit experts also varied. In most studies, opinions were sought from a sample of study investigators,26,30 members of the general population or a membership list or mailing list or by using a search strategy to identify authors of published studies on a particular subject. 134,136 Experts who were listed as members of relevant specialty organisations were recruited either through random selection222 or by contacting all members of the list (e.g. members of the Royal College of Psychiatrists). 237 Convenience sampling (e.g. from patients attending an outpatient clinic227 or those who noticed an advertisement requesting participants)124,224 was also used. For face-to-face meetings, participants could be recruited from among conference delegates. 238 Of the remaining studies, either experts were gathered from special interest groups (e.g. conference attendees or a relevant subcommittee) or the method of expert recruitment was not reported. The number of experts whose views were sought varied from 5239 to 1584,240 although these data were not always reported. 9

A survey was the most common mechanism by which an opinion was sought, although interviews or face-to-face meetings were also used occasionally. A combination of survey and face-to-face methods or survey and interview-based approaches was also used. 26 In one case the method involved a face-to-face meeting and additional elicitation although the method was not reported. 239 When studies involved face-to-face meetings, the method of deriving consensus for a resulting important difference value once opinions had been elicited was either not explicitly specified43 or was derived using established techniques for such processes (e.g. nominal group technique). 238 Specified processes commonly involved some kind of quantitative mechanism for eliciting opinions to summarise agreement, for example through voting,230 or the compilation of individual attendees’ scoring results for a set of paper patients. 241 Splitting the number of attendees into groups was used in some face-to-face meetings. 229,238 If survey methods were used alongside face-to-face meetings, survey responses could be gathered before the meeting,238 during the meeting234 or after the meeting. 241

For group-based decision-making it was necessary to establish a threshold for acceptable consensus, and this varied from situation to situation. Delphi processes were often used in studies using an opinion-seeking approach. Most of the studies using survey and face-to-face methods for eliciting an important difference used a Delphi process (normally two ‘rounds’),134–136,241 although it should be noted that most of the studies doing so were written by the same group of authors. A Delphi process was also used by some of the studies that had used only a survey method for seeking opinions (although again many had been undertaken by the same authors). 242–245 Several studies used an 80% consensus threshold,246–248 or a value very close to this level (‘all but two’ out of nine experts). 136 An example opinion-seeking method study in which the Delphi approach was used is described in Table 4.

Other studies using a survey varied in terms of the complexity of the data collected. Some simply asked what value on a particular instrument respondents would consider to be clinically significant. 232,237,249 Items or criteria can also be ranked in terms of their importance in helping participants form their judgement(s) regarding the extent to which a particular value represents an important difference. 232,240 Opinion could also be sought by study investigators on participants’ preferences for particular hypothetical scenarios. For example, in the study by Allison and colleagues,223 respondents (obesity patients entering clinical trials) were asked to indicate on a VAS how much of an increased risk of a serious adverse event they would tolerate in order to lose a variety of expressed proportions of their current body weight (5%, 10%, 20%, 30%). Trade-off and standard gamble methods were also used in some of the studies using an opinion-seeking method that had interviewed participants (either face-to-face or by telephone). 16,222,224,225,227,228,250 Another study using interview techniques226 asked participants (patients and clinicians) to rate the clinical importance (on a five-point Likert scale) of specific goals for the patient, rank the five most important from among these rated goals and indicate how much improvement would be required to consider that the patient had achieved a meaningful difference in quality of life or functional capacity. Man-Son-Hing and colleagues251 used a flip chart and booklet during interviews to graphically assist participants in conceptualising risk. In some instances, rather than being asked about their expectations of change over time with regard to a particular disease, treatment and outcome measure, respondents (clinicians and triallists) were asked about thresholds that would influence their practice. 26,30,47 Fayers and colleagues26 asked surgeons about their expectations regarding a particular treatment. Similarly, the CHART trials provide an example of eliciting opinion regarding equivalence and a realistic difference for a survival outcome to determine the target difference in a Bayesian framework (although the sample size was calculated using the conventional approach). 30 Latthe and colleagues45 developed a distribution for doctors’ prior beliefs about the effectiveness of a particular treatment, using a survey method for seeking opinion. Oremus and colleagues47 additionally enquire in their study not only about thresholds for equivalence between treatments, but also about thresholds for the likelihood of a treatment to meet first-line therapy standards. The analysis of experts’ views also varied in complexity. For example, Rider and colleagues247 developed a large number of definitions of clinical improvement using CART analysis and logistic regression, to compare with values from an opinion-seeking method, identified (using diagnostic accuracy methodology) to determine a smaller number of definitions that had good (80–85%) sensitivity and specificity. These definitions were returned to the experts who were asked to rate their face validity. Similar approaches were used by Ruperto and colleagues248,252 and Giannini and colleagues. 246

Summary of opinion-seeking studies

A total of 81 studies were found that used the opinion-seeking method; 21 also used another method (see Combination of methods for details). Most commonly, opinion-seeking methods had been applied in the fields of rheumatology, respiratory-related outcomes or mental health, although there was a wide range of specialties noted and studies did not always relate to a specific clinical specialty (e.g. studies focusing on the opinions of triallists). 253

Practical considerations for use of the opinion-seeking method

One advantage of the opinion-seeking method is the ease with which it can be carried out (e.g. through a survey). Eliciting an opinion within a trial context has been carried out utilising a Bayesian framework,26,30 which can be more informative but also more difficult to implement. The estimate may not be representative of the desired population or a future trial and values may differ between experts and stakeholders (e.g. patients and clinicians). For non-survey approaches one major issue for implementation is the sample size. With face-to-face meetings there will be an upper limit on the number of people whose opinion can be sought if consensus on a value or a small range of values is the goal of the meeting. Results may not be generalisable to a larger population than the one envisaged244 and the sample may not be representative of the general population. 223 Face-to-face meetings may be influenced by group processes. 134 If this is felt to be a problem, a Delphi method could be implemented through a survey to compile the views of multiple participants anonymously. 242 However, survey methods may be vulnerable to typical difficulties such as low response rate, missing data and technical problems if online and so careful planning is required. 41 The Delphi method itself allows for variation in implementation of how consensus is reached.

The nature of the expertise itself may influence the results, as being involved in a trial could bias an expert’s judgement of the estimates they provide (e.g. being too enthusiastic about a treatment’s possible effects). 26 Indeed, the specialty may influence values26,254 and the views of participants may change over time. 239 In addition, a pre-existing definition of an important difference (e.g. anchor-based estimate) can be used to help participants formulate their own estimate. 250 The wording of the question is important as it needs to be clear and specific enough to prevent different participants making varying assumptions about the situation they have been presented with. 255 Interview-based methods (telephone or face-to-face) might affect estimates124 and questions may need to be rephrased,231 although these approaches allow for clarification, unlike survey formats. Trade-off approaches (commonly used for interview-based opinion-seeking approaches) may be problematic as hypothetical scenarios may not be adequate predictors of real behaviour,124 may be considered too artificial226 and may be difficult to comprehend,223 particularly if participants are not used to understanding risk concepts. Describing risk to participants in different ways can have an impact on the resultant estimate. 228

Brief description of the pilot study method

A pilot study may be used as a method to determine a relevant value for the target difference in situations in which there is little evidence, or even experience, to guide expectations on an appropriate value for the target difference. One definition of a pilot study is running the intended study in miniature before conducting the actual trial. 256 Similarly, a Phase II study in the regulatory drug setting could be viewed as a pilot for the Phase III study. 257 By carrying out a pilot study that recruits an identical, or similar, population to the one that is expected to be recruited to a future trial, the resulting data can be used by triallists to estimate parameters for a sample size calculation for the main trial. This can be the effect size or one of the parameters needed (e.g. SD for a continuous outcome or the event proportion for a binary outcome/survival outcome). Pilot studies are typically small in size.

Variations on the pilot study method

The simplest approach would be to use the observed effect in the pilot study as the target difference in a RCT. Such an approach has been criticised. 44 One paper258 modified this approach by calculating the baseline SD of the Roland–Morris Questionnaire (RMQ) in the pilot study, which was used in the sample size calculation for a future trial for a given difference in the outcome.

The study by Salter and colleagues259 used a similar approach but adjusted the estimate for the SD for the uncertainty by using the upper limit of a 90% CI of the variance to allow for the imprecision. This study also inflated the calculated sample size to account for similar levels of attrition as seen in the pilot study. This need to adjust the pilot study SD estimate was shown in the study by Browne,42 which looked at the likelihood of actual power being greater than or equal to planned statistical power in a future trial. A simulation was carried out for combinations of various pilot sample sizes (5, 10, 30, 50 and 100), mean differences between trial arms (10, 30 and 75), which gave standardised effects of 0.1, 0.3 and 0.75, respectively, with a prespecified SD of 100, and planned statistical power of 80% and 90%. Using one of five upper confidence limits (50%, 60%, 70%, 80% or 90%) as an estimator of SD (as opposed to the SD point estimate), Browne showed that the probability of the actual power exceeding the planned power can be substantially increased by taking account of uncertainty around the pilot estimate. A hypothetical example involving treatment for nephropathy was provided to illustrate the findings. Wang and colleagues257 undertook a simulation study of the expected sample sizes for a Phase III trial based on the observed difference in effect size from a Phase II trial (which was considered a pilot for this Phase III study). By simulating values for the Phase II sample size (50, 100 and 200) with planned power of 80%, and setting α to 0.025, this study showed the effect of using the point estimate of the effect size from the Phase II trial compared with using the lower limit of both 1 and 2 SDs from the point estimate (the standardised effect for a Phase III trial was expected to be lower than that of a Phase II trial). This study also anticipated that there would be a threshold effect size for a Phase II study below which investigators would not proceed to a Phase III trial because it would require such a large sample size as to be unfeasible.

Summary of studies using the pilot study method

Six studies used a pilot study method. Five of these studies used this method alone42,44,257–259 and one used it in combination with another method260 (see Combination of methods for details). Two studies258,259 had conducted pilot interventions designed to reduce pain, whereas in the study by Browne,42 a hypothetical example involving renal patients was provided to illustrate the method used. The sample sizes of the simulated pilot studies ranged from 542 to 200257 and for the real pilot studies ranged from 12258 to 160. 260

Practical considerations for use of the pilot study method

Pilot studies are not merely useful for sample size calculations but can also provide additional information on feasibility and highlight challenges to participant recruitment that may need to be taken into account in a sample size calculation. 259 There are practical difficulties in undertaking a pilot study that may limit the relevance of the result (e.g. changes in the population, intervention or outcome definition and timing of measurement);258 however, the most problematic is the inherent uncertainty in the study results because of the small sample size. The estimate of the effect size will likely be very imprecise. Use of a pilot study to inform either the SD or the control group event proportion is more useful although the imprecision of these values should also be acknowledged. 257,260 Strictly speaking, an internal pilot, that is, assessing data from the participants in the first stage of the study, cannot be used to specify the target difference, or a related sample size component (e.g. control group event proportion), although it could be used as part of an adaptive strategy to preserve the prespecified target difference. A pilot study is most useful when it can be readily and quickly (e.g. rapid recruitment and short outcome follow-up) conducted.

Brief description of the review of evidence base method

The RoEB method summarises existing evidence in order to identify an important or a realistic difference for a specific treatment comparison. The optimal implementation would be to meta-analyse results for the outcome of interest based on the studies found using a defined search method. An alternative approach is to review existing evidence for a specific outcome of interest to determine a difference that can be viewed as important.

Variations on the review of evidence base method

The most common approach in included studies261–268 involved implementing a prespecified strategy for reviewing the evidence base for either a particular instrument or a variety of instruments for an important difference. These studies identified, from the results across all studies reporting the same outcome, a plausible value for an important difference for the instruments or outcomes under consideration. Cranney and colleagues264 reviewed existing studies reporting an important difference for particular outcomes in osteoporosis. In addition, they distinguished between ‘individual’ and ‘group’ important differences. For this reason they retained a selection of osteoporosis RCTs that had been identified from their review search strategy to determine the level of an important difference that had been used in the RCTs. Blumenauer and colleagues261 used the method of identifying studies reporting an important difference for a particular outcome measure, although did not provide a conclusive value for an important difference in summary. However, the values identified using the RoEB method for particular outcome measures were then compared with the results of trials that reported those outcomes.

It was also common to review the level of observed differences (similar to the ‘group’ important differences used by Cranney and colleagues264). Of note, Revicki and colleagues267 considered differences for mortality. Several studies required included studies to have reported sufficient detail to enable effect size estimates to be calculated by the reviewers. In the studies by Johnston and colleagues,269 Thomas and colleagues270 and Woods and colleagues,271 this was done to determine the sample size of a future study. The last study used Cohen’s h statistic for a binary outcome (see Combination of methods). The study by Thomas and colleagues270 reported meta-analysis of data from identified reviews within the evidence on the subject of interest (sports medicine outcomes). The effect size used in the power calculation was therefore a pooled effect size. Pooled standardised effects were also used by Woods and colleagues. 271 The study by Johnston and colleagues269 used the effect sizes identified from reviewing the evidence base to derive a sample size calculation for a future RCT (i.e. a realistic target difference). Others had used a similar approach in reviewing the observed effect size, but did not explicitly use the findings in a sample size calculation. 272,273 One study by Schwartz and colleagues274 found some evidence to dispute Norman and colleagues’275 concept of the universality of half a SD. The study by Nietzel and colleagues276 calculated an effect size with reference to a normative population. RCI calculations comparing treated groups with a normative population were also found. 273,276–278

Julious23 provided worked examples of using a pre-existing meta-analysis result to determine the target difference for a new study and also considered the associated uncertainty around the estimated pooled effect. An example of assessing uncertainty regarding the SD based on previous studies is also considered. Extending this general approach, Sutton and colleagues28 derived a distribution for the effect of treatment from the meta-analysis, from which they then simulated the effect of a ‘new’ study; the result of this study was added to the existing meta-analysis data, which were then reanalysed. This process was repeated a large number of times until a power calculation was made using the proportion of times out of the total number of simulations that the null hypothesis was rejected. Implicitly this adopts a realistic difference as the basis of the target difference. Additionally, the primary analysis of the trial should involve the other studies given that the study was justified in this way. Zanen and Lammers279 reviewed studies in order to derive a CoV statistic to use in the sample size calculations for an equivalence trial.

Summary of review of evidence base studies

A total of 28 studies were found that used the RoEB method; six of these also used another method (see Combination of methods for details). Several studies262,264,268–270,275 reported using a systematic approach to reviewing the evidence base.

Practical considerations for use of the review of evidence base method

Reviewing the existing evidence base is valuable as it provides a rationale for choosing an important or realistic value for the target difference in the sample size calculation of a future clinical trial. However, an estimate identified from the existing evidence may not necessarily be directly appropriate for the population under consideration in a future clinical trial, that is, the generalisability of the identified values may be questionable because of their methodological risk of bias, the intervention implementation or the population studied. 263,264,280 The extent to which it is possible to account for these factors depends on the level of detail provided in the previous studies. In addition, publication bias is a relevant issue to consider, particularly for reviews of the evidence base that do not consider alternative sources of information besides searching databases of published evidence sources. 267 The formal use of meta-analysis results to determine the sample size28 implies that a future study will also be analysed using the current evidence and, arguably, that if a further study is published during its conduct then the sample size should be updated. A review of studies that have determined an important difference is also possible although each individual study (use of anchor method) would have the practical issues mentioned earlier for that type of method. For both approaches, similar to the use of a pilot study, the imprecision in the estimate (whether the target difference or an associated parameter, for example the SD) needs to be considered. Although a meta-analysis is likely to have greater precision than a pilot study, imprecision is still an important consideration.

Brief description of the standardised effect size

The SES method calculates the effect size on a standardised scale. For a continuous measure this is usually simply the difference in means (either between two groups of people or between two time points in the same group of people) divided by an appropriate SD (Cohen’s d effect size). The SD is usually the pooled SD of the groups when a comparison between groups (e.g. treatment) is being made, whereas the SD of either the first time point or the change score is often used for within-group comparisons (e.g. before and after treatment). The magnitude of this standardised effect is then used to assess whether an important difference has occurred. Guidelines of 0.2 SDs, 0.5 SDs and 0.8 SDs are widely used for interpreting the magnitude of the effect, and therefore an observed effect can be interpreted according to these values. Alternatively, by specifying the SD, an effect on the original scale of a particular magnitude can be determined. For binary outcomes a variety of SES metrics (e.g. odds and risk ratios or an absolute difference in event proportion) exist. A hazard ratio or absolute difference in event proportion can be used for a survival (time-to-event) outcome.

Variations on the standardised effect size

Three type of standardised effects have been defined for a continuous outcome: between groups, within group and compared with a reference population. 281 The between-groups standardised effect is the situation that mirrors the specification of the target difference in a parallel groups trial, in which a difference between two groups is tested. In the simplest case (equal group sizes) it is calculated for an outcome, x, as:

The formula can be modified for uneven sample sizes by calculating the (weighted) pooled SD.

In most of the studies identified, the use of standardised effects related to calculation of a within-group effect. For the within-group standardised effect, most commonly in the literature on important differences, the standardised effect is often used to assess changes in health over time for a quality-of-life measure. The most common situation is to measure subjects’ health on a particular outcome measure at two time points: baseline (before treatment) and follow-up (after treatment). First, the group mean change in score from baseline to follow-up (or equivalently the difference in the mean scores at the two time points) for individuals in a sample is calculated. This mean change in score is divided by the SD of the baseline score: 282–290

The SD used varied. An example is given in Box 4 and shows the possible impact of using different SDs to calculate the SES. Other authors divided the mean change in score by the SD of the change in score. 291–294 The resulting measure is sometimes called the standardised response mean (SRM):9

A number of studies used both approaches. 295–298 Howard and colleagues43 used a different formula for the SD of the change scores, which accounted for within-person correlation.

An alternative is to use the pooled SD (SD_pooled) of the two time points. 289,299 In the simplest case (i.e. equal group sizes) it is calculated as:

In some studies, the SES was of the mean change between groups rather than within one group over time. 289,290,300–302 A further variation on the choice of SD was described by Horton,303 who compared the difference in means between two groups, divided by the largest SD value.

The study by Andrew and Rockwood299 calculated the SES as a ‘modified Cohen’s d’. A standard Cohen’s d is produced using the pooled SD except that this SES is ‘corrected’ for the (Pearson’s) correlation between the baseline and follow-up scores:

Fredrickson and colleagues304 utilised a more complex within-group SES for a crossover design. In this study each subject received all four different treatments over four time periods, and subjects’ health was measured at the end of each time period. The authors used the following formula (Dunlap’s d) to calculate the SES between each active treatment group and the placebo group at each follow-up time point:

where t_c is the SE (from a paired t-test) of the mean difference in the change of health between baseline and follow-up between the treatment groups, r is the correlation between the pairs of measures and n is the number of observations. Two studies compared study results with data from a reference population who served as normative data. 273,305 Harris and colleagues305 subtracted the mean score of a reference population from the baseline and follow-up mean scores of a sample and then divided by the SD of the reference population score. The SES is calculated as x₁ – x₂, where

A lower value for the score used in this study indicated a better outcome. A similar method was used by Rajagopalan and colleagues. 306 They divided the study sample into those with and those without a condition of interest. They then divided the mean difference between these two groups by the ‘population SD’. However, it is not clear whether they are referring to a reference population (and, if so, which one) or all of the subjects in their study. The similarity of such an approach to the RCI distribution method (described in Distribution method) should be noted.

One other study used a similar approach for a categorical outcome with repeated measures. 289 The study authors used the difference in proportions between baseline and follow-up. SES metrics are commonly used for binary (e.g. odds ratio, risk ratio) and survival outcomes (e.g. hazard ratio) in medical research307 and a similar approach can be readily adopted although this review identified no studies that formally carried this out. A doubling or halving of a ratio is sometimes seen as a marker of a large relative effect. 308

Summary of standardised effect size studies

A total of 166 studies were found that used the SES; 129 also used another method (see Combination of methods for details). The papers reported the results from studies in a range of clinical areas, including, but not limited to, Alzheimer’s disease, back pain, cardiac surgery, insomnia, schizophrenia, stroke and osteoarthritis. The most common outcome of interest was quality of life. Other outcomes of interest included disability, functional ability, goal attainment, speech function and cognitive ability. In considering change, some studies often used a clinician to assess subjects’ health,282,287,290,293,301,303 whereas others used the subjects’ self-assessment292,297,302,309–311 or both296,299,312,313 (not necessarily for the same instrument). The view of the parents of children who were being treated was often used in studies in paediatric care along with clinical and/or child assessment. 314–316 Similarly, Rockwood and colleagues312 used the combined judgement of the patient and the caregiver as well as the clinician’s assessment. Basoglu and colleagues291 used one instrument that was administered by a clinician, but it was unclear whether the patient or the clinician completed other instruments.

Many studies used the method in order to inform the magnitude of a difference in an outcome that can be viewed as important. A standardised effect was often used to compare the health of different groups of people, either people with a disease with population norms273 or people with a condition of interest with those without the condition. 306 Some papers were interested in assessing the effect of a treatment on health or comparing the effects of different treatments. 289,301,304,305 The study by Fredrickson and colleagues304 included repeated measures over time, as well as different doses of a medication, as the authors were also interested in using standardised effects to investigate when changes in health occurred. Four papers specifically calculated effect sizes to inform the design of a future study. 283,300,302,317 The larger a standardised effect, the fewer the number of subjects required to detect a difference between groups or time points. 283 Pyne and colleagues302 went a step further and calculated effect sizes for different subgroups of people to assess whether any of the instruments performed better in certain subsets of people. Fredrickson and colleagues304 also made specific reference to the use of standardised effects for calculating required sample sizes, but did not perform any such calculations. van der Putten and colleagues287 made reference to the link between how responsive an instrument was (as measured by a SES) and the sample size required for, and power of, a study to detect a statistically significant result. In addition, Rockwood and colleagues’312 study used the SES from an early study to determine the sample size necessary to power their RCT appropriately.

Overwhelmingly, studies used the values suggested by Cohen, whether explicitly or implicitly, where 0.2, 0.5 and 0.8 represent a small, medium and large effect respectively. Some studies had not merely utilised a SES as a post hoc indication of the magnitude of change for responsiveness purposes, but instead had selected a value a priori for the effect size level that would denote clinically meaningful change, and then calculated the corresponding value for the outcome of interest that would correspond to this level of change. 290,299,318–322 The a priori value for clinically meaningful change varied between studies from 0.2 SDs290,318 to 0.5 SDs,299,319,322 0.8 SDs321 and both 0.4 SDs and 0.8 SDs. 320 Alternative values for small, medium and large SESs (0.1, 0.5 and 1.0) have been suggested although without any identifiable usage. 323 However, several of the studies that had used multiple methods, of which one was a SES approach, had used alternative proportions of the SD for small (minimal) change, including a quarter of a SD (0.25 SDs),82,114,324 a third of a SD (∼0.33 SDs)325,326 or even 0.3 SDs. 327 Occasionally, studies considered multiple levels as being of interest for interpretation of an important difference, for example both 0.5 SDs and 0.2 SDs. 328–331 Sometimes larger values were used, and two studies332,333 used 1 SD as an effect size cut-off, of which the study by Hurst and Bolton332 had also used 0.6 SDs. When studies did not cite Cohen specifically, many still interpreted their SES estimates according to his guidelines. 334 Even when the values were not explicitly stated, most studies had based their interpretation of the effect on Cohen’s criteria, concluding that effect sizes of change were ‘large’,335 ‘moderate’336,337 or ‘small to moderate’. 338

Practical considerations for use of the standardised effect size

The main benefits of using the SES method are that it can be readily calculated and it allows comparison across different outcomes, conditions, studies, settings and people. The latter is possible because all of the differences are translated into a common unit. Such an approach is frequently used in meta-analyses to summarise findings across studies. 304 In addition, it is relatively easy to calculate effect sizes for published studies, provided that they have reported sufficient information. 283,304 However, different combinations of values can produce the same value on the standardised scale. For the standard Cohen’s d statistic, different combinations of mean and SD values produce the same SES estimate; for example, a mean and SD of 5 and 10, respectively, as well as values of 2 and 4, respectively, give a standardised effect of 0.5 SDs. A larger effect on the original scale can therefore have the same effect on the standardised scale when there is greater variance (e.g. when there is more variability in response because of a more heterogeneous population or more imprecise measurement). As a consequence, specifying the target difference as a SES alone, although sufficient in terms of sample size calculation, can be viewed as an insufficient specification in that it does not define the target difference in the original scale. Furthermore, it can be difficult, if not somewhat futile to try, to explain why different effect sizes are seen in different studies, for example whether these differences are due to differences in the outcome measures, interventions, settings or subjects in the studies. Therefore, as Haymes and colleagues283 recommend, a comparison of effect sizes should be interpreted cautiously. Nevertheless, given its ready application, the SES approach (with Cohen’s interpretation) is useful when no more pertinent evidence is available; anecdotally, ‘large’ effects are not commonly observed in many settings.

Different SES metrics have been used for continuous (e.g. Cohen’s d and Dunlap’s d), binary (e.g. odds ratio) and survival (hazard ratio) outcomes;271,307,339 however, overall, Cohen’s d is by far the most commonly used metric. Although SESs are commonly used for binary outcomes (e.g. odds ratio, risk ratio) and survival outcomes (hazard ratio) we did not identify any studies that proposed similar guidelines/approaches for their use in determining an important difference, although they could in principle also be used in a similar manner. Cohen’s h was the only binary SES metric271 that was identified for which such a formal assessment of standardised effect magnitude was made, although this will commonly be carried out informally (e.g. halving or doubling of risk/odds as being an important effect)308 when conducting sample size calculations or assessing the impact of relationships in epidemiology for metrics such as the odds, risk and hazard ratios. The vast majority of the literature relates to within-group SESs for a continuous outcome. The SD used varied between studies (i.e. between baseline, final and change score values); this undermines comparability as standardised effects based on within-person changes will tend to be larger. Although a variety of SES metrics exist for binary outcomes, the commonly used metrics (such as the odds ratio) do not uniquely identify the sample size and need to be considered in conjunction with the control group proportion. SESs have been used to compare the responsiveness, discriminatory ability and sample size requirement of instruments. They can also be used to compare treatment effects or assess changes in health over time.

Standardised effect sizes are relatively easy to calculate and are objective and there is a widely accepted guideline for the interpretation of the effect sizes for a continuous outcome. Effect sizes of < 0.2 are considered ignorable, effect sizes of ≥ 0.2 to < 0.5 are considered small, those ≥ 0.5 to < 0.8 are considered moderate and those ≥ 0.8 are considered large, based on Cohen’s guidelines. 339 Approximate equivalent values for the odds ratio can be calculated using Cohen’s cut-offs giving 1.44, 2.48 and 4.27 for a small, medium and large effect respectively. 340 Corresponding risk ratio values vary according to the control group event proportion. 307 However, these guidelines may not be appropriate in some situations and were not intended by Cohen to become default values for all purposes. Cohen suggested that, in settings outside of a laboratory or when investigating new research areas, only small effects are to be expected. 290 Some attempts to assess the generalisability of the guidelines have been undertaken285,341,342 and provide some reassurance. Gordon and colleagues293 suggested that, without the use of an established instrument as a reference criterion (or anchor) for assessing changes, it is still a judgement call as to what constitutes a clinically important difference in a particular situation. Both Cheung and colleagues300 and Pyne and colleagues302 made use of an anchor to categorise the health or changes in health of their study subjects. Dumas and colleagues282 called for further research using an external criterion to investigate clinically meaningful changes. Matza and colleagues284 specifically state that, although an effect size can be used to assess how small a change in health an instrument can detect, another method (such as an anchor or distribution method) is needed to determine what would be considered a MCID. However, this view is not universally accepted. For example, Konst and colleagues301 specify that a large effect size indicates a clinically important change in health.

Several studies measured the health of the same people over time although had not adjusted for the correlation between measurements recorded at baseline and measurements recorded at follow-up when calculating effect sizes. 299,304 Making this adjustment will lead to a different magnitude of standardised effect, with larger correlations leading to larger effects unless the change score SD is used. 299,304 As noted above, sometimes the SD at baseline is used. 283–288 Fredrickson and colleagues304 argue that this approach is ‘flawed as it assumes that differences in performance between subjects [on different treatments] at baseline will provide a reliable estimate of differences within subjects over time thereby potentially reducing the estimate of the effect size [of one treatment compared with another]’. With regards to target difference, the SES based on a change score is not directly comparable to that based on a two-group parallel study. Only one study was identified that used a SES approach for a categorical outcome although it was analysed as a continuous outcome. 289 Occasionally, studies did not give any details of how they calculated the standardised effect, apart from sometimes referencing other papers. 282,309,317,318,343

A clear disadvantage of using the SES method to specify a target difference is that the SES is dependent on the group means and SDs. Different populations will have different means and SDs285 and therefore it is reasonable to expect effect sizes, and therefore what might be called small, medium and large effects, to vary between comparisons according to interventions, outcomes and conditions. Pyne and colleagues302 speculated that, because experimental studies tend to have stricter inclusion criteria and greater control over any interventions than observational studies, changes in health might be larger in experimental studies. Experimental studies may overestimate actual effect sizes or provide an underestimate. 302 When determining a target difference for a RCT based on the SES method, the SD used should refect the anticipated RCT population and, as far as possible, the statistical analysis.

Summary of methods used within studies combining more than one method

Of the included studies, 216 used more than one method. A summary of how often each method was used with one or more additional methods is given in Table 5. Details of the numbers of studies that used each of the different combination of methods are available in Table 6.

Superiority, equivalence and non-inferiority

Most studies are based on testing for superiority; they are designed to address whether an outcome differs (in either direction) between two interventions. Although this chapter primarily focuses on superiority studies, we note that the specification of a target difference also occurs for equivalence and non-inferiority studies, that is, an equivalence margin needs to be determined within which the interventions may be viewed as having an equivalent (or non-inferior) outcome.

Stand-alone study or evaluated in conjunction with other evidence

Although existing evidence is routinely available and often informally used,48 a RCT is conventionally designed as a stand-alone experiment. A decision needs to be made whether existing data should be formally incorporated into the study sample size calculation. 28,428 It has been argued persuasively that a new trial should be designed on the basis of available evidence and once the trial is completed the data it provides should be incorporated into the evidence base. 27 There are, however, good reasons for desiring a stand-alone methodologically robust study that is large enough to provide a definitive answer. It is reasonable to have concerns about reproducibility, applicability and consistency in methodology, which cannot always be fully addressed within a meta-analysis framework. 434,435

Very large studies designed as a single ‘final’ study are sometimes referred to as ‘mega’ or ‘large simple’ trials,435,436 for example the CRASH trials, which investigated the impact of treating trauma patients with corticosteroids437 and tranexamic acid. 438 However, there is a role for conducting a study that, when combined with current evidence, would have sufficient statistical precision to answer the research question. 48 The desired level of precision may not be feasible within a single study; additionally, substantial evidence may already exist. It should be noted that following this approach to the calculation of the sample size carries the obligation that the trial is analysed on the same basis, that is, the main analysis should include all of the available evidence (existing evidence and the new trial). Another possible scenario is when multiple trials could be potentially conducted simultaneously with a view to formally combining once all data are available (see Review of evidence base method for further discussion). 439 Nevertheless, if existing data are used in the sample size calculation, the basis of the target difference will need to consider the importance of any difference deemed realistic as opposed to solely achieving a statistical difference (of any magnitude). 28

General considerations

Statistical sample size calculation is not an exact science. 433 First, investigators typically make assumptions that are a simplification of the anticipated analysis. For example, the impact of controlling for prognostic factors is very difficult to predict and even though the analysis is intended to be adjusted (e.g. when randomisation has been stratified or minimised)440 the sample size calculation is often based on an unadjusted analysis. Second, the calculated sample size can be very sensitive to the values of the inputs; in some circumstances a relatively small change in the value of one of the inputs (e.g. the control group event proportion for a binary outcome) can lead to substantial change in the calculated sample size. For example, a 10% difference between groups requires 313 per group for 80% power at the two-sided 5% significance level441 if the assumed control group level is 80%; however, if the control group value used is 60% then 408 per group are needed. A further consideration is that sample size calculations are conducted a priori to provide reassurance that study results will be informative. However, the value used for one of the inputs (e.g. control group event proportion) may not accurately reflect the actual value that will be observed in the study. Although reassessing the assumed control group proportion or SD is not unusual during the conduct of a large study (whether utilising a formal statistical approach or a more simple check based on interim data), it is clearly not possible to know the final value until completion. Failure to fully understand this concept is reflected in the incorrect use of ‘post hoc’ sample size calculations. 428 Performing a sample size calculation to determine what would be the likely result (i.e. statistical power at a particular significance level given the other inputs) when the actual analysis result is available is not sensible.

Statistical approaches

Different statistical approaches can be adopted to calculate the sample size. The vast majority of trials adopt the same basic methodology based on the Neyman – Pearson approach. 24,428 In essence, this approach involves adopting a statistical testing framework and calculating the sample size required given the specification of two statistical parameters (the power and significance level – see below for definitions). Although different types of outcomes, study designs (e.g. cluster randomised trials442) and analyses require modification of the formula, the general approach is similar across study designs. Other statistical approaches for defining the required sample size are Fisherian, Bayesian and decision-theoretic Bayesian approaches, along with a hybrid of both the Bayesian and Neyman–Pearson approaches. 24,443,444 Although the sample size calculation process is different under a Bayesian statistical framework with regards to specification of the statistical aim, there is some similarity. A range in the posterior distribution of an outcome in a Bayesian statistical framework is often used in a similar manner to the equivalence limit in an equivalence trial under the Neyman–Pearson method. 24,29 Health economic-based methods developed since the late 1990s tend to follow a Bayesian approach19 and sometimes also utilise a decision-theoretic framework38 (see Chapter 2 and Description and guidance on the use of individual methods for specifying the target difference). Bayesian methods readily allow incorporation of prior beliefs (which can be based on empirical evidence and/or opinion). 24 However, methods other than Neyman-Pearson are rarely used, as confirmed by a review of RCT sample size calculations, which identified only the Neyman–Pearson approach in use in a sample of over 200 studies. 22 As noted above, this document focuses on the Neyman–Pearson approach except when the method for specifying the target difference necessitates an alternative approach (see, for example, Health economic method).

To calculate the sample size for a superiority trial under the Neyman–Pearson approach, a compromise is required between the possibility of being misled by chance into concluding that there is a difference (in the intervention effect) when there is not and the risk of not identifying a genuine difference. Once all of the inputs have been specified, the required sample size can be determined. In a more complex situation, a simple formula may not be available and a simulation study of the hypothesised true intervention effect, under the assumptions and using the proposed statistical analysis, is needed to calculate the required sample size. The standard approach involves specifying a null hypothesis of no difference and seeking evidence to reject the null hypothesis in favour of the alternative hypothesis (there is a difference in either direction). Two types of errors can be made (type I and type II) as noted in Chapter 1. Commonly used values are α = 0.01 or 0.05 for a type I error (and commonly referred to as a 1% or 5% statistical significance level) and β = 0.1 or 0.2 for a type II error (90% or 80% power to detect a difference of the size specified) under the Neyman–Pearson statistical approach. Without much exception a two-sided significance level is adopted as it is rare that the possibility of a difference in both directions is not conceivable. Even when a one-sided test approach is adopted, such as would be the case for a non-inferiority study, it is conventional to adopt the same one-tailed significance as if it were a two-sided test (i.e. a one-sided 2.5% significance level for a non-inferiority study corresponds to a two-sided 5% test such as would be used for a superiority study). 23 Once the two statistical criteria (the significance level and the power) are set and the statistical test to be conducted is chosen, the sample size is determined depending on the magnitude of difference to be detected (‘target’ difference) and associated component of the outcome (i.e. SD for a continuous outcome). The target difference is the magnitude of difference (intervention effect) that the RCT is designed to investigate in the primary outcome.

Choosing a primary outcome

A variety of factors need to be considered when choosing a primary outcome. First, in principle, the primary outcome should, as noted above, be a ‘key’ outcome such that knowledge of its result would help answer the research question. For example, in a RCT comparing treatment with eye drops to lower ocular pressure with observation for patients with high eye pressure (the key treatable risk factor for glaucoma, a progressive eye disease that can lead to blindness), loss of vision is a natural choice for the primary outcome. 445 However, it would clearly be important to consider other outcomes (e.g. side effects of the drug). Nevertheless, knowing that the eye drops reduced the loss of vision due to glaucoma would be a key piece of knowledge. In some circumstances, the preferable outcome will not be used because of other considerations. In the above glaucoma example a surrogate might be used because of the time it takes to measure any change in vision noticeable to a patient. The primary outcome should always be a key outcome for the comparison being undertaken. Second, consideration is needed about the ability to measure the chosen primary outcome reliably and routinely within the context of the study. Missing data are a threat to the analysis of any study and RCTs are no different. The optimal mode of measurement may be impractical or even unethical. The most reliable way to measure eye pressure (intraocular pressure) is through manometry;446 however, this requires invasive eye surgery. Subjecting participants to clinically unnecessary surgery for the purpose of a RCT is clearly not ethical without very strong mitigating circumstances, particularly as an alternative, even if less accurate, way of measuring intraocular pressure exists. Furthermore, in the context of manometry, an informed consent process would lead to a substantial number of people not consenting to the surgery required for the manometry, if not the study overall. Third, the outcome needs to be one for which an appropriate difference can be detected with an achievable sample size. There are different bases for determining this difference, although whatever value is used, it will need to lead to an achievable sample size. Obtaining a sufficient sample size is in turn dependent on there being sufficient potential participants, a practical time frame for the outcome measurement and interventions that will provide an answer before any technology becomes obsolete, and which can be achieved with the financial and other resource constraints faced. When planning a study it is not unusual for a number of outcomes to be considered as potential primary outcomes, and a judgement has to be made as to which best meets these criteria. The development and use of core outcome sets across studies will help ensure that, even when a core outcome is not used in the sample size calculation, it is collected and reported. 447

In some circumstances more than one primary outcome may be used. 428 For example, more than one outcome may be used to cover the different aspects of the purported difference. 6 This is less common in a regulatory setting as formal statistical adjustment for the use of two (or more) outcomes may be required through adjustment of the significance level for multiple comparisons. 428 Such adjustments are often overly conservative and can negatively impact on the ability to detect a difference for any of the chosen primary outcomes. Alternatively, the sample size calculation may also be conducted in a way that ensures sufficient precision for a secondary outcome. 448 For clarity, the remainder of this chapter will be based on the premise of only one primary outcome, although we note that more than one might be appropriate in which case a judgement about the value of statistical correction for multiple outcomes will have to be made. 428

Types of outcome

The sample size formula varies depending on the outcome and the intended analysis. The most common outcomes are binary, continuous and survival (time-to-event) outcomes; continuos outcomes are typically assumed to be normally distributed, or at least ‘approximately’ so, for ease and interpretability of analysis and for the sample size calculation. All three types are considered below; we do not consider other types of outcome measure although we note that ordinal, categorical and rate outcomes can be used, for which a more complex analysis and corresponding sample size calculation approach may be needed. From a purely statistical perspective, a continuous outcome should not be converted to a binary outcome (e.g. converting a quality-of-life score to high/low quality of life) and the sample size calculation based on the resultant binary outcome; such a dichotomisation would result in less statistical precision and lead to a larger sample size being required. 449 If viewed as necessary to aid interpretability, the target difference (and corresponding analysis) used in the continuous measure can also be represented as a dichotomy in addition to being expressed on its continuous scale. Some authors, although acknowledging that this should not be routine, would make an exception in some circumstances when a dichotomy is seen as providing a substantive gain in interpretability even if it is at a loss of statistical precision. 450

Key points for using the anchor method to specify the target difference

• Suitable for continuous (or ordinal) outcome.

• Anchor implementation is critical; for example, the perspective and anchor adopted.

• Particularly suited to quality-of-life measures.

• The magnitude of the difference can be sensitive to the population group (e.g. ceiling/foor and disease severity effects may exist).

• Use of the most common anchor approach implies that a within-person (important) difference can be applied though a between-person approach is also possible.

Key points for using the distribution method to specify the target difference

• Suitable for a continuous (or possibly ordinal) outcome.

• Use of the distribution method (i.e. measurement error approach) is of limited merit due to its weak justification of an ‘important’ difference.

• A simple range or levels approach should be a last resort if no more informative methods can be used and only when the outcome has clear meaning.

Key points for using the health economic method to specify the target difference

• Allows a comprehensive approach to the value of a RCT; in particular, the costs of the intervention and its comparator and research can be considered in conjunction with possible benefits and consequences of decision-making. The flexible modelling framework allows any type of outcome to be incorporated.

• The perspective adopted is critical – the viewpoint and values that are used to determine the scope of costs and benefits incorporated into the model structure.

• Uncertainty around inputs can be substantial and extensive sensitivity analyses will likely be needed. Some inputs (e.g. time horizon) will be particularly challenging to specify as well as appropriately representing the statistical relationship of multiple parameters. These could also be based on empirical data and/or expert opinion.

• This is a resource-intensive and complex approach to determining the sample size.

• Unlikely to be accepted as the sole basis for study design at present despite intuitive appeal. Patients and clinicians may be resistant to the formal inclusion of cost into the design and thereby the primary interpretation of studies. Expressing the difference in conventional way is likely to be necessary as it is more intuitive to stakeholders and also furthers the science of interventions. It could provide additional justification for conducting a large and expensive trial (e.g. when there is a small effect and/or events are rare).

Key points for using the opinion-seeking method to specify the target difference

• Allows for varying degrees of complexity of the scenario (e.g. consideration of related effects or impact on practice) and any outcome type (binary,47,451 continuous242,451 and survival30).

• The perspective is critical – whose opinions are being sought.

• A realistic and/or important target difference can be sought.

• A target difference that takes into account other outcomes and/or consequences (e.g. a target difference that would lead to a health professional changing practice) or focuses exclusively on a single outcome can be sought.

The presentation of the scenario is important. Ideally a mechanism to confirm/probe the initial response will be used. 463

Key points for using the pilot study method to specify the target difference

• There is a need to assess the relevance of the pilot study to the design of a new RCT study. Some down-weighting (whether formally or informally) may be needed according to the relevance of the study and methodology used. For example, a Phase II study should be used to directly specify a (realistic) target difference for a Phase III study only if the population and outcome measurement are judged to be sufficiently similar.

• Helpful for estimating outcome components such as variability of a continuous outcome (or control group rate for a binary outcome) although the estimation of the target difference is typically imprecise because of a small sample size.

• This approach can be used in conjunction with another method (e.g. using an opinion-seeking method to determine an important difference) to allow full specification of the target difference.

Key points for using the review of evidence base method to specify the target difference

• It should be based on a systematic search of available evidence.

• It can be used for any outcome type (including continuous, binary, ordinal and time-to-event outcomes).

• A choice must be made whether an important and/or a realistic difference is sought.

• A number of issues need to be considered when assessing an observed difference:

  • Is the evidence available directly relevant to the research question at hand (PICOT assessment)?466

  • Is the existing evidence of a robust nature? Are there multiple studies available and were they conducted in a methodologically robust manner? What was the risk of bias?467

  • Is the outcome of interest fully reported? Individual patient data are seldom available and reporting of outcome is often selective. 468

• Determination of a realistic (target) difference can, and when possible should, be based on a systematic review and associated meta-analysis of RCTs, although imprecision in the estimate needs to be considered.

• The use of prior evidence can be formalised through simulation of the impact of a new study on the meta-analysis result,28 although this implies that a particular analysis will be conducted and the new study will be analysed alongside the current evidence.

Key points for using the standardised effect size to specify the target difference

• The SES for a continuous outcome should be calculated as the difference between groups divided by the appropriate SD. For a parallel group trial, the SD will typically be an estimate of the (common) group SD, which corresponds to an unadjusted analysis of the final scores; the SD of the within-person change score could be used when an analysis of change scores is planned. The benefit of removing within-person variance, such as through an analysis which adjusted for the baseline value, can also be incorporated when the correlation can be estimated. 473

• A SES from a before-and-after treatment study is unlikely to be representative of that achievable in a treatment study, particularly when two active treatments are compared.

• Use of Cohen’s criteria of interpretation is difficult to justify, although widespread. Modifications to this effect size scale have been suggested (see Chapters 2 and 3). 474 For example, pragmatic trials475 are generally accepted to have smaller effects than more efficacy-focused studies. The SES may differ in magnitude between clinical areas and outcomes and when the standard treatment is very effective.

• Changes in the variability (e.g. population spectrum) for a continuous outcome can result in a different standardised effect even though the mean difference remains the same. It is important that an estimate of the variability is also specified and that the sample is similar to the anticipated RCT population. For a binary outcome, the target difference (whether a relative or an absolute difference) should be considered in conjunction with the control group event proportion.

• It is most appropriate as a fall-back option if other more context-relevant methods for specifying the target difference cannot be used. 455,476

Reporting items for the randomised controlled trial protocol

• State any divergence from the conventional approach.

• State the primary outcome (and any other outcome that the study sample size calculation is based on), or state why there is not one.

• Reference the formula/simulation approach if standard binary, continuous or survival outcome formulae are not used. 23,430 The primary analysis should be stated in the statistical analysis section.

• State the values used for statistical parameters (e.g. significance level and power).

• State the underlying basis used for specifying the target difference:

  • an important difference as judged by a stakeholder

  • a realistic difference based on current knowledge or

  • both an important and a realistic difference.

• Express the target difference according to the outcome type:

  • Binary – state the target difference as an absolute and/or a relative effect, along with the intervention and control group proportions. If both an absolute and a relative difference are provided, clarify if either takes primacy in terms of the sample size calculation.

  • Continuous – state the target mean difference on the natural scale, the common SD and the SES (mean difference divided by the SD). It is preferable to also provide the anticipated control group mean even though it is not required for the sample size calculation.

  • Survival (time-to-event) – state the target difference as an absolute and/or relative difference; provide the control group event proportion, and the intervention and control group survival distributions; additionally, the planned length of follow-up should be stated. If both an absolute and a relative difference are provided, clarify if either takes primacy in terms of the sample size calculation.

• Explain the choice of target difference – specify and reference any formal method used or relevant previous research.

• State the sample size based on the assumptions specified above (for a survival outcome, the number of events required should also be stated). If any factors (e.g. allowance for loss to follow-up) that alter the required sample size are incorporated they should also be specified along with the final sample size.

Reporting items for the randomised controlled trial results paper

• State any divergence from the conventional approach.

• State the primary outcome (and any other outcome that the study sample size calculation is based on), or state why there is not one.

• State the values used for statistical parameters (e.g. significance level and power).

• Express the target difference according to the outcome type:

  • Binary – state the target difference as an absolute and/or a relative effect, along with the intervention and control group proportions. If both an absolute and a relative difference are provided, clarify if either takes primacy in terms of the sample size calculation.

  • Continuous – state the target mean difference on the natural scale, the common SD and the SES (mean difference divided by the SD). It is preferable to also provide the anticipated control group mean even though it is not required for the sample size calculation.

  • Survival (time-to-event) – state the target difference as an absolute and/or relative difference; provide either the intervention and control group event proportions or the intervention and control group survival distributions; additionally, the planned length of follow-up should be stated. If both an absolute and a relative difference are provided, clarify if either takes primacy in terms of the sample size calculation.

• State the sample size based on the assumptions specified above (for a survival outcome, the number of events required should also be stated). If any factors (e.g. allowance for loss to follow-up) that alter the required sample size are incorporated they should also be specified along with the final sample size.

• Reference the trial protocol for further details.