Avoiding and identifying errors in health technology assessment models

Article history

The research reported in this issue of the journal was commissioned by the HTA programme as project number 07/54/01. The contractual start date was in May 2007. The draft report began editorial review in March 2009 and was accepted for publication in November 2009. As the funder, by devising a commissioning brief, the HTA programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

None

Copyright statement

© 2010 Queen’s Printer and Controller of HMSO. This journal is a member of and subscribes to the principles of the Committee on Publication Ethics (COPE) (http://www.publicationethics.org/). This journal may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NETSCC, Health Technology Assessment, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2010 Queen’s Printer and Controller of HMSO

Background

The National Institute for Health and Clinical Excellence (NICE) in England and Wales and similar structures elsewhere are required to make health-policy decisions that are relevant, evidence-based and transparent. Decision-analytic modelling is recognised as being well placed to support this process. 1 The key role that models play is, however, reliant on their credibility. Credibility in decision models depends on a range of factors including the coherence of the model with the beliefs and attitudes of the decision-makers, the decision-making framework within which the model is used, the validity of the model as an adequate representation of the problem in hand and the quality of the model.

The predominant software platform for Health Technology Assessment (HTA) models is the spreadsheet. In studies of generic spreadsheet system development, there is abundant evidence of the ubiquity of errors and some evidence of their potential to impact critically on decision-making. The European Spreadsheet Risks Interest Group (EuSPRIG) maintains a website recording research on risks in spreadsheet development and on public reports of errors (www.eusprig.org/ accessed 20 February 2009). In 1993, Cragg and King2 undertook an investigation of spreadsheet development practices in 10 organisations and found errors in 25% of the spreadsheets considered. Since 1998, Panko3 has maintained a review of spreadsheet errors and this suggests that spreadsheet error rates are consistent with error rates in other programming platforms, and when last updated in 2008 error rates had not shown any marked improvement.

To date, there has been no formal study of the occurrence of errors in models within the HTA domain. However, high-impact errors have been recorded, perhaps the earliest being the case of the sixth stool guaiac test, where Neuhauser and Lewicki4 estimated an incremental cost of $47 million per case of colorectal cancer detected. Brown and Burrows5 subsequently identified an error in the model and generated a comparatively modest corrected estimate. The grossly inflated incorrect figure has, however, subsequently been used (and is still in use) by authors including Culyer6 and Drummond et al. 7,8 in their seminal texts to demonstrate the importance of incremental analysis in health economics. The failure in this case was traced back to an error in the interpretation and subsequent modelling of diagnostic test characteristics. In a study in 2000 investigating the quality of models used to support a national policy-making process, Australia’s Pharmaceutical Benefit Advisory Committee (PBAC) reported that around 60% were flawed in some way and 30% demonstrated problems in the modelling. 9 Although it is tempting to think that things may get better over time, a recent repeat retrospective analysis reported in 2008 demonstrated that in fact the situation had appeared to deteriorate with 82% (203 out of 247) of models reviewed being considered by PBAC to be flawed in some respect. 10

Errors in software implementing mathematical decision models or simulation exercises are a natural and unavoidable part of the software development process. 11 This is recognised outside the HTA domain and accordingly there exists research, for example in the computer sciences field and in the operational research field, in model testing and verification. In contrast, the modelling research agenda within HTA has developed primarily from a health-services research, health economics and statistics perspective and there has to date, been very little attention paid to the processes involved in model development. 12 The extent of this shortcoming is reflected in the fact that guidance on good practice either acknowledges the absence of methodological and procedural guidance on model development and testing processes, or, makes no reference to the issue. 13–17

A wide range of strategies could be adopted with the intention of avoiding and identifying errors in models. These include improving the skills of practitioners, improving modelling and decision-making processes, improving modelling methods, improving programming practice and improving software platforms. Within each of these categories there are many further options, so improving the skill base of practitioners might include: the development and dissemination of good practice guides, identification of key skills and redesign of training and education, sponsoring of skills workshops, and determination of minimum training/skill requirements. These interventions again might impact across the whole range of disciplines involved in the decision support process, including information specialists, health economists, statisticians, systematic reviewers and not to forget, modellers themselves. Outside the HTA domain, Panko identifies a wide range of initiatives for best practice in spreadsheet development, testing and inspection, including such things as guidelines for housekeeping structures in spreadsheets. 18 However, Panko also goes on to review the evidence on the effectiveness of these interventions and identifies first a lack of high-quality research on effectiveness and second that what evidence does exist is at best equivocal. 18 This, taken together with the evidence that models are not improving over time, indicates that caution should be exercised in recommending quick-fix, intuitively appealing solutions.

In the absence of an understanding of error types and the causes of errors, it is difficult to evaluate the relative merits of such initiatives and to develop an efficient and effective strategy for improving the credibility of models. This study seeks to understand the nature of errors within HTA models, to describe current processes for minimising the occurrence of such errors and to develop a first classification of errors to aid discussion of potential strategies for avoiding and identifying errors.

Aim and objectives

The aim of this study is to describe the current comprehension of errors in the HTA modelling community and to generate a taxonomy of model errors, to facilitate discussion and research within the HTA modelling community on strategies for reducing errors and improving the robustness of modelling for HTA decision support.

The study has four primary objectives:

1. to describe the current understanding of errors in HTA modelling, focusing specifically on

   1. types of errors

   2. how errors are made

2. to understand current processes applied by the technology assessment community for avoiding errors in development, debugging models and critically appraising models for errors

3. to use HTA modellers’ perceptions of model errors together with the wider non-HTA literature to develop a taxonomy of model errors

4. to explore potential methods and procedures to reduce the occurrence of errors in models.

In addition, the study describes the model development process as perceived by practitioners working within the HTA community, this emerged as an intermediate objective for considering the occurrence of errors in models.

Methods overview

The project involved two methodological strands:

1. A methodological review of the literature discussing errors in modelling and principally focusing on the fields of modelling and computer science.

2. In-depth qualitative interviews with the HTA modelling community, including:

   1. – Academic Technology Assessment Groups involved in supporting the NICE Technology Appraisal Programme (hereafter referred to as Assessment Groups)

   2. – Outcomes Research and Consultancy Groups involved in making submissions to NICE on behalf of the health-care industry, (hereafter referred to as Outcomes Research organisations).

In-depth interviews with HTA modellers

Literature review search methods

At the outset the scope and fields of literature relevant to the study were not known. The purpose of the searches was twofold; first exploratory searches in advance of the interviews were undertaken to inform the development of the scope and content of the in-depth interviews, second the searches after the interviews were undertaken to expand the discussion on key issues raised by the interview analysis and to place the interview data in the context of the broader literature.

An iterative approach was taken to searching the literature. This is recognised as being the most appropriate approach to searching, in the case of reviews of methodological topics, where the scope of relevant evidence is not known in advance. 22 Based on information-seeking behaviour models23 the exploratory search strategy used techniques including author searching, citation searching, reference checking and browsing.

The scoping search was undertaken to establish the volume and type of literature relating to modelling errors within the computer science and water resources fields both identified as potentially rich fields. Highly specific and focused searches were carried out in Computer and Information Systems Abstracts and Water Resources Abstracts.

The interviews and scoping search of the literature identified the need for further reviews including: first, a topic review of error avoidance and identification; second, a review of model verification and validation; and third a review of error classifications and taxonomies. The searches were undertaken in February 2008 and January 2009. The search strategies and sources consulted are detailed in Appendix 2.

Validity checking

The report was subjected to internal peer review by a qualitative researcher, mixed methodologist, modellers not involved in the interviews and a systematic reviewer. The final draft report was distributed to all interviewees to obtain their feedback on whether the findings are representative of their views, hence providing respondent validation. 24

The organisation of the interview sample

This section briefly details the background, expertise, experience and working arrangements of the 12 individuals included in the interview sample.

Overview

This chapter presents a descriptive analysis of individual respondents’ perceptions of the model development process. The purpose of eliciting information concerning current model development processes within the interviews was twofold. First, it allows for the development of an understanding of the similarities and variations in modelling practice between modellers that might impact on the introduction of certain types of model error. Second, the elicitation of interviewee’s personalised view of the process was used to form the structure of the remaining portion of the interview, with the model development charts therefore forming a map against which specific types of model error could be identified and discussed.

The descriptive analysis was undertaken according to the coding scheme, and examined six key stages of the model development process: (1) understanding the decision problem; (2) use of information; (3) implicit/explicit conceptual modelling; (4) software model implementation; (5) model checking activities; and (6) model reporting. It should be noted that these six stages were not specified a priori, but rather they emerged from the qualitative synthesis process. A seventh theme concerning iterations between stages of model development was also analysed. The descriptive synthesis of the model development process is presented together with an inferred stylised model development process describing both the range and diversity of responses.

Understanding the decision problem

For all 12 interview participants, the first stage in the model development process involved developing an understanding of the decision problem. Broadly speaking, this phase involved a delicate balancing/negotiation process between the modeller’s perceived understanding of the problem, the clinical perspective of the decision problem, as well as the clients’ needs and the decision-makers’ needs. This phase of model development may draw on evidence, including published research and clinical judgement, and experience within the research team. In some instances the research question was perceived to follow directly from the received decision problem, whereas other respondents expressed a perception that the research question was open to clarification and negotiation with the client/decision-maker. In this sense the research question represents an intellectual leap from the perception of the decision problem. One respondent noted that this aspect of model development may not be a discrete step which is finalised before embarking on research:

The whole process starts at the point where we are thinking about a project but it will continue once a project has actually started, almost until the final day when the model is finalised that process will continue…we will not finalise the structure of the model and the care pathway perhaps until quite close to the end of the whole project because we will be constantly asking for advice about whether or not we have it correct or at least as good as we can get it.

[I8]

Similarly, another respondent highlighted that the decision problem may not be fixed even when the model has been implemented, indicating that modelling may have a role in exploring the decision problem space. Hence, for instance, modelling can answer questions such as ‘Is technology A economically attractive compared to technology B’ or ‘Under what circumstances is A economically attractive compared to B’. These are fundamentally different decision problems however the decision-makers may change their focus throughout the course of an analysis and are dependent on the outcomes of the analysis.

…commonly at the analysis stage, you have the opportunity or there’s the possibility of changing the question that you’re asking. So, if the results of the model are in any way unexpected or in fact, if they’re in any way insightful, then you might want to refine or amend the outcome or the conclusion you’re going to try and support. So if we start off with an analysis of a clinical trial population that is potentially cost-effective, in the course of the model, you find that the product is particularly appropriate for a subgroup of patients, or there are a group of patients where the cost-effectiveness is not attractive, then we’d refine the question.

[I6]

Several interviewees noted the importance of understanding the decision problem and defining the research question appropriately; in particular, one respondent indicated the fundamental importance of understanding the decision problem before embarking on the implementation of any model:

…it all stems from what is the decision problem, to start off with…no matter what you’ve done, if you’ve got the decision problem wrong…basically it doesn’t matter how accurate the model is. You’re addressing effectively the wrong decision.

[I7]

Conceptual modelling

Nine of the twelve interviewees either alluded to, or explicitly discussed, a set of activities involving the conceptualisation and abstraction of the decision problem within a formal framework before implementing the mathematical model in a software platform.

Use of information in model development

For many of the interviewees, the use of evidence to inform the model development process was broader than the generation of systematic reviews of clinical effectiveness. The use of evidence used to determine the model structure was also commonly discussed. Further, the use of evidence in informing the model was seldom restricted to a single stage in the development process; for several respondents, evidence was used to understand, shape and interpret all aspects of model development.

Implementation model

All respondents discussed a set of activities related to the actual implementation of a mathematical model on a software platform. Such activities either involve the transposition of the prespecified soft conceptual model framework into a ‘hard’ quantitative model framework involving various degrees of refinement and adaptation, or the parallel development of an implicit conceptual model and explicit implementation model.

Model checking

Given the extent of discussion around model checking during the interviews, a detailed analysis of current methods for checking models is presented (see Chapters 6 and 7). Model checking activities varied and appear to occur at various stages in the model development process. One interviewee suggested that model checking begins once the model has been built; however, for other respondents, the checking process happens at various distinct stages. One respondent indicated that model checking activities are undertaken throughout the entire model development process. Whereas, subject to certain nuances in the precise methods employed, much of the discussion around current model checking activities focused on testing and understanding the underlying behaviour of the model and making sure that it makes sense:

…it’s about finding out what the real dynamics of the model are so what are the key parameters that are driving the model, where do we need to then emphasise our efforts in terms of making sure our data points are precise? What are the sensitivities, in which case what subgroups might we need to think about modelling? Discussing how the model is behaving in certain situations. What the likely outcome is. If the ICER is near to the critical thresholds for the decision-maker then where we need to pay attention if the ICER is way off to the side, if you’ve got an output that’s dominated or if you have got an ICER that’s half a million pounds it seems likely that unless the model is wildly wrong the decision is going to be fairly clear cut. So if you are operating at the decision-making threshold how precise and careful you have to be and what kind of refinement you need therefore to build into the model.

[I4]

Other model checking activities included examining the face validity of the model results in isolation or in comparison to other existing models, internal and clinical peer review, checking the input data used in the model and checking premodel analysis can replicate the data, and checking the interpretation of clinical data. Many of these approaches involve a comparison of the actual model outputs against some expectation of what the results should be; the interview data were not clear how such expectations were formed.

Either the intuition or the modelling or the data is wrong and we tend to assume that it is only one of them…I think you tend to assume that once they’re [clinical experts] not surprised by the thing then, that means you have got it right.

[I5]

Two respondents indicated a minimalist approach to model checking activities:

So we do just enough…just enough but not as much as you’d want to do.

[I10]

…it depends whether you’re error searching for what I term significant errors or non-significant errors, I probably would stop and the non-significant errors that are probably in a larger percentage of models than people would like to know would remain.

[I2]

Reporting

The majority of respondents referred to model reporting as the final step in the model development process. This stage typically involved writing the report, translating the methods and results of the model and other analysis; and engaging with the decision. Although little of the interview content concerned this phase of the process, two important points were raised within the interviews. First, one respondent highlighted that the reporting stage may act as a trigger for checking models; where the results are unexpected or queried by internal or external parties. The second point concerned the importance of the interpretation of the model by decision-makers and other process stakeholders:

I think the recommendation we would make is we need to pay more attention to understanding how our models are understood and how we present them, a lot of work that can be done in model presentation.

[I4]

…reportable convention standards can be very important in ensuring everyone has a clear view of what’s being said. There are ways in which model outputs can be more transparently depicted and the key messages conveyed to users more clearly, often some can get lost in the text or hidden away somewhere either intentionally or unintentionally.

[I4]

Synthesised model development process

Figure 1 shows a stylised representation of the model development process generated through a synthesis of all interview transcripts and from the charted descriptions of the model development process elicited from the interviewees. The stylised model development process is comprised of five broad bundles of activities including; understanding the decision problem, conceptual modelling, implementing the model within a software platform, checking the model, and finally engaging with the decision. It should be noted that these activities do not perfectly match the coding scheme used to undertake the Framework analysis. The general remit of these sets of activities is as follows.

FIGURE 1.

Stylised model development process showing key iterations between sets of activities.

Understand the decision problem This phase involves immersion in research evidence, definition of research question, engaging with clinicians, engaging with decision-makers, understanding what is feasible, understanding what the decision-maker wants/needs and engaging with methodologists.

Conceptual modelling It is a commonplace to state that all mathematical models are built upon a conceptual model. At its most banal it is impossible to implement a model without first conceiving the model. The key issue in the decision-making endeavour and reflected in the modelling process is communication: specifically the process of developing a shared understanding of the problem. Conceptual modelling is the process of sharing, testing, questioning and agreeing this formulation of the problem; concerned with defining the scope of a model and providing the inputs to the process of systems analysis and design associated with defining a solution to the problem. The scope of a model deals with defining the boundary and depth of a model, to identify the critical factors that need to be incorporated in the model. Models are necessarily simplifications of our understanding of a problem. The conceptual model allows a description of one’s understanding of the system that is broader than the description of the system captured in the model. An implemented model will therefore be a subset of a conceptual model. This hierarchy allows simplifications represented in the model to be argued and justified. This process of simplification is the process of determining relevance.

The arrow leading from ‘understanding the decision problem’ to ‘implementation model’ reflects the views of three interviewees who implied that they either do not build a conceptual model or that conceptualisation and implementation are simultaneous activities.

Implementation model This phase involves the implementation of the model within a software platform.

Model checking This stage includes all activities used to check the model. This includes engaging with clinical experts to check face validity, testing extreme values, checking logic, checking data sources etc. A detailed description of current model checking activities is presented in Chapters 6 and 7.

Engage with decision This phase concerns the reporting and use of the model with and by the decision-maker.

Activities concerning the use of evidence, peer review and other clinical consultation may arise within any or all of the five model development phases.

Several points are of note in the interpretation of the diagram. First, as the diagram has been developed through a synthesis of all interview data, respondents did not adopt a uniformly standard model development process, and respondents discussed different activities to varying degrees of detail, the lines of demarcation between main sets of activities were not entirely clear. Second, the diagram is intended to be descriptive rather than prescriptive; it describes current model development processes rather than a normative judgement about how models should be developed. The dashed arrows in Figure 1 indicate significant iterations between key sets of model development activities. It should also be noted that each individual phase is likely to involve iterations within sets of activities, e.g. there may be several versions of an implementation model as it is being developed.

The iterative nature of developing mathematical models is well documented. The interview data suggest six key points of model iteration within the development process. The first substantial iteration (marked as number 1 on Figure 1) concerns developing an understanding of the decision problem; this may involve iterations in terms of striking a balance between what the analyst believes are the decision-makers needs, what is feasible within the project resources and negotiations therein.

The second notable iteration (marked as number 2 on Figure 1) involves looping between the development of an explicit conceptual model and its implementation in a software platform. Several respondents highlighted the inevitable existence of a circular loop between these two sets of activities whereby the intended conceptual model structure is redefined in light of limited evidence available to populate that structure, and whereby the evidence requirements are redefined in light of the intended model structure.

The interview data suggest two further loops relating to iterations from the model checking stage to revising or refining the conceptual and/or implementation model (marked as numbers 3 and 4 on Figure 1). The interview data suggest that any of the large number of checking activities currently undertaken have the capacity to result in backwards iterations to earlier steps.

The fifth key iteration of note (marked as number 5 on Figure 1) concerns the use of an existing model for multiple decision problems. In such instances, there is a loop between the final and first stages of the process. Two respondents highlighted examples of this iteration: one in terms of the use of ‘global models’ for use in different decision-making jurisdictions, and one in terms of the ongoing development of independent models across multiple appraisals. As noted earlier (see Use of information in model development), several respondents discussed the use of existing models as the basis for developing a model structure. In such circumstances, there is a danger that adopting an existing model developed by another party and applying this to a new decision problem could effectively represent a loop back to a conceptual model without a comprehensive understanding of the decision problem.

The final iteration (marked as number 6 on Figure 1) loops from model checking to understanding the decision problem. The three respondents who highlighted that implementation and conceptualisation activities occur in parallel all indicated the possibility of revising and rebuilding the model from scratch. The same suggestion was not indicated by the remaining nine respondents. At the other extreme, one respondent indicated that there was no significant backwards iteration once the implementation model was under way, i.e. the conceptual model was not amended once agreed.

Discussion of model development process

The analysis presented within this chapter highlights considerable variation in modelling practice between respondents. This may be in part explained by the apparent variations in expertise, background and experience between the respondents. The key message drawn from the qualitative analysis is that there is a complete absence of a common understanding of a model development process. Although checklists and good modelling practice have been developed, these perhaps indicate a general destination of travel without specifying how to get there. This represents an important gap and should be considered a priority for future development.

The stylised model development process presented in Figure 1 represents a summary of all interviewees’ stated approaches to modelling. It should be noted that the diagram does not entirely represent the views of any single individual, but is sufficiently generalisable to discriminate between the views of each interviewee; in this sense, any modeller should be able to describe their own individual approach to model development through reference to this diagram. As noted above, one particularly apparent distinction that emerged from the qualitative synthesis was the presence or absence of explicit conceptual modelling methods within the model development process. Although it is difficult to clearly identify a typology on these grounds from a mere 12 interviewees, the qualitative data are indicative of such a distinction. Related to this point is the issue of the a priori specification of the model design and analysis, which represents the leap from the conceptual model to the implementation model. Some respondents did discuss the preparation of materials which go some way in detailing proposed conceptual models, platforms, data sources, model layout, checking activities and general analytical designs; however, the extent to which these activities are specified before implementation appears to be limited.

A common feature of modelling critiques that is reflected in the interviews is the statement that different teams can derive different models for the same decision problem. This arises from the largely implicit nature of existing conceptual modelling processes described by the interviews. It is suggested that focusing on the process of conceptual modelling may provide the key to addressing this critique. Furthermore, the quality of a model depends crucially on the richness of the underlying conceptual model. Critical appraisal checklists of models all currently include bland references to incorporating all important factors in a disease yet do not address how these ‘important factors’ can be determined. It is the conceptual modelling process that is engaged with identifying and justifying what are considered important factors and what are considered minor or irrelevant factors for a decision model.

The conceptual modelling process is centrally concerned with communication and this can be supported by many forms including, conversation, meeting notes, maps (causal, cognitive, mind etc.), skeletal or pilot models. However, care needs to be exercised in using methods that focus on the problem formulation rather than formulation of the solution. 26

The description of the model development processes adopted by the interviewees within this chapter is directly used to inform the development of the taxonomy of model errors (see Chapter 5) and to provide a context for analysing strategies for identifying and avoiding model errors (see Chapters 6 and 7).

Overview

This chapter presents the description of key dimensions and characteristics of model errors and attempts to draw a boundary around the concept of ‘model error’, i.e. ‘what is it about a model error that makes it an error?’. The descriptive analysis highlighted both overt and subtle variations in respondents’ perceptions concerning what constitutes a model error. In seeking to explain, assign meaning and draw a boundary around the concept of model error, the respondents identified a variety of factors including:

• non-deliberate and unintended actions of modellers in the design, implementation and reporting of a model

• the extent to which the model is fit for purpose

• the relationship between simplifications, model assumptions and model errors

• the impact of error on the model results and subsequent decision-making.

The above factors are discussed in this chapter, the interview findings are then placed in the context of the literature on model verification and validation.

Intention, deliberation and planning

The interview data strongly suggested a general consensus that model errors were unintended, unplanned or non-deliberate actions; instances whereby the modeller would have done things differently had they been aware of a certain aspect or underlying characteristic of the model; or instances whereby the model produced unexpected or inconsistent results. The unintended actions of the modeller and the non-deliberate nature of the modellers’ actions when developing models appeared to be central to the interviewees’ definitions of model error.

Extent to which the model is fit for purpose

A commonly cited dimension of a model error concerned the extent to which the model could be considered fit for purpose. In particular, respondents highlighted that a model could be unfit for its intended purpose in terms of the underlying conceptual model as well as the software implementation model, so representing distinct errors relating to model design as well as errors relating to the execution of the model. Respondents implied a relationship between the concept of model error and the extent to which a model is fit for purpose, suggesting that the presence of model errors represents a threat to the model’s fitness for purpose, whereas the development and use of a model that is unfit for its intended purpose is a model error in itself.

Several interviewees drew reference to concepts of verification and validity as a means through which to define the concept of model error. Where interviewees used these terms there was a consensus regarding their definition, with verification referring to the question ‘is the model right?’ and validation referring to ‘is it the right model?’. Although there was general consensus that the term model error included issues concerning the verification of a model, there was less clarity about the relationship between model errors and validity. For instance, problems in the conceptualisation of the decision problem and the structure of the model were generally referred to as ‘inappropriate’ rather than ‘incorrect’, yet nonetheless these were described as model errors by some respondents.

I think it’s probably easier to know that you have made an error of verification where there’s clearly a bit of wrong coding or one cell has been indexed wrongly or the wrong data has been used for example which wasn’t intended. So those are all kinds of error that are all verification type errors. Errors in the model rather than it being the wrong model which is a much bigger area the area of validation and there people can argue and it can be open to debate.

[I4]

Further, it should be noted that one interviewee indicated that errors affecting the model validity, that is errors in the description of the decision problem and conceptual model potentially dwarfed technical errors that affected its verification.

…my belief is that if you spend…the majority of your time, as I say, getting the decision problem right, making sure you understand data and how that data relates to the decision problem and decision model, that…the errors of programming itself, those ones which are much more minor.

[I7]

A repeatedly stated dimension of model error that emerged from the interviews was the failure to adhere to best practice or failure to accord to current conventions. Alternative viewpoints included the notions of failure to do ‘the best you can do’ or failure to do ‘what you should do’. It was noted that in some instances this may be the result of a lack of skills. Further complexity was apparent in terms of the selection of modelling method and its relationship to model errors:

I don’t think I would class [it] as an error when someone says that…they developed this hugely complex…kind of Markov-type model, and then said they’d much rather have done it as a simulation.

[I11]

Simplification versus error

The qualitative synthesis suggested mixed views between the concepts of model assumptions and model errors. For example, one respondent appeared to hold a strong view that all inappropriate assumptions were model errors, highlighting the importance of the perspective of the stakeholder (or model user) in distinguishing between the two concepts:

…One man’s assumption is another man’s error…

[I3]

Other respondents highlighted a fine line between the necessary use of assumptions, in model simplification and the introduction of errors into the model:

If you have an inaccuracy in the model that has no, or a trivial effect, then it’s not obvious to me where that stops being a simplification that you’ve made to make the decision problem tractable, and where it becomes an error,…something that’s inaccurate and we wish to avoid…that point would be where it starts to cause a risk that the model is giving a wrong answer, or is not addressing the question that you wanted it to.

[I6]

…it’s very rare that the model is completely accurate in that it does exactly what it says it’s meant to be doing. And it’s never the case that the model is completely accurate in that it’s a full and complete picture of the world. And I’m very comfortable with that, because the whole point of doing your model is that it’s a simplification, and that you’re trying to extract a simplified view of the world that teaches you something new that wasn’t obvious from the data that you start with.

[I6]

Further to this discussion, one respondent highlighted a distinction between assumptions and errors by examining the underlying reason for the use of the assumption:

So you have to say, ‘Well, to what extent are the choices being made being made on the grounds of evidence, and how much were they on the grounds of convenience?’ Now if they’re made on the grounds of convenience and in addition to that you know, from my point of view they are mathematically illiterate, then that is an error, to me. They are unsustainable.

[I3]

Contrary to this viewpoint, one respondent suggested that unreasonable assumptions and the overoptimistic interpretation of evidence did not constitute a model error because of their deliberate intent. An alternative suggestion for distinguishing between assumptions and model errors concerned the perspective of the decision-maker; assumptions were errors if the decision-maker does not ‘own’ them, does not feel comfortable with them, or cannot support them upon a wider appraisal of evidence.

Impact of error on the model results and subsequent decision-making

A number of interviewees suggested that the existence of model errors is inevitable; that it is only important to ensure that the model is free from significant errors which have an important influence on the model outputs and the resulting policy decision.

…I think it’s rare to find any model without an error in it somewhere…I think we’re deluding ourselves if we think…that there are no errors in our models…I think [that] even if you identify one, that doesn’t mean there aren’t others, and if you don’t identify one that doesn’t means there are none.

[I5]

An error is something that markedly changes the ICER regardless of whether it changes the conclusion.

[I2]

The problem is…can you really cope with that error if someone’s making a policy decision based on that outcome?

[I4]

One interviewee made the interesting observation that the impact of an error should be considered relative to the overall uncertainty in results:

…if it changes [the incremental cost-effectiveness ratio] from £8266 to…£8267 technically it’s an error, the question is, those technical errors will be absolutely dwarfed by the uncertainty in the efficacy of the drug, so I wouldn’t consider them an error.

[I2]

This begs the question how wrong does something have to be before it becomes a model error?

Consequently, for some interviewees, this led to a very broad boundary around what would be defined as an error:

…it’s a mistake at any point in the…appraisal process, from the initial specification of a decision problem through to the assumptions, the parameter inputs and the analysis that goes behind that, through to the final implementation and programming of an excel-based model. And even then, an error in the interpretation of the results therein.

[I7]

Placing the interview findings in the context of literature on model verification and validation

In 1979 the Society for Modelling and Simulation International (SCS) defined the distinction between model verification and validation. 27

• Model validation is defined as ‘substantiation that a computerised model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model’.

• Model verification is defined as ‘substantiation that a computerised model represents a conceptual model within specified limits of accuracy’.

The definitions of model validation and verification provided by the HTA modellers interviewed in this study were best captured by the description that verification refers to the question ‘is the model right?’ and validation refers to the question ‘is it the right model?’. These definitions are consistent with the SCS definitions. Although the interviewees descriptions might lack the apparent rigour of the SCS definitions, it should be noted that this rigour is perhaps somewhat illusory because in both cases the SCS definitions simply recast the problem as one of defining what constitutes a satisfactory range of accuracy.

In searching for rigorous approaches to validation and verification of Operational Research models many authors draw reference to epistemology and philosophies of science. In 1967 Naylor et al. 28 rehearsed the opposition between rationalism and empiricism and proposed a three-stage approach to model verification based explicitly upon merging these principles; it should be noted that at this early stage the terms verification and validation were used interchangeably. Naylor et al. suggest that rationalism is represented in the initial search for a set of postulates regarding the components, variables and functional relationships that constitute a model and specifically that the search for face validity of a model by its accordance to expert judgement is a rationalist perspective on validity. However, the authors consider these initial postulates only as ‘tentative hypotheses’ on the behaviour of the system and suggest that these postulates should be verified empirically, and that this verification constitutes an empiricist perspective on validity. The third stage of validation suggested by Naylor et al. , consists of testing the ability of a model to predict, retrospectively or prospectively, the behaviour of a system and represents an application of a positive economics perspective on model validity. The three stages outlined above are closely reflected in the HTA interviewees’ description of the model development process outlined in Chapter 3 and indeed in the discussion of simplification versus error contained earlier in this chapter.

This early literature on model verification and validation focuses on modelling as a knowledge creation process. This perspective has led to validity being defined with reference to a model’s ability to predict observable phenomena. In certain methodological work this epistemological perspective has led to the definition of statistical criteria for the goodness of fit of a model as a measure of its validity. 29,30 None of the HTA modellers interviewed pursued this perspective of validity.

In 1993 the European Journal of Operational Research31 published a special issue on validation in modelling, with the purpose of raising debate on this topic. The papers presented in this special issue expand upon the epistemological roots of model validation methodology. A key characteristic of operational research modelling is the focus, not on creating knowledge, but rather on supporting decision-makers. Moreover, supporting decision-makers working within a broad social and political context, working with a range of criteria and with particular approaches to the decision-making process. This focus on supporting decisions underlies the importance of the pragmatic, instrumentalist or utilitarian philosophies for model validation. Under these pragmatic approaches the usefulness of a model, that is its success in action, is its measure of validity. Déry et al. 32 acknowledge Raitt33 and Rorty34 as first explicitly bringing this utilitarian approach to the discussion of model validation. The interviewees’ principle focus on fitness for purpose as opposed to statistical accuracy of models reflects strongly this instrumentalist viewpoint on validity, all of the HTA modellers interviewed were consistent in adopting this perspective.

Kleindorfer et al. 35 in 1998 reviewed the philosophical underpinnings of model validation and extended the discussion to the implications of Hermeneutics as propounded by Gadamer36 and more recently Bernstein. 37 Hermeneutics recognises that objectivity is unachievable and suggests that meaning is created through intersubjective communication. It is characterised by a description of rationality that is historically situated and practical, involving choice, deliberation and judgement. Kleindorfer’s reading of Hermeneutics is used to provide a basis for the primacy of interaction between the modeller and client in developing mutual understanding of a model. It is this mutual understanding that establishes a model’s significance and its warranty. Kleindorfer et al. propose a framework for model validation based upon the analogy of a judicial system, where the model builder is free to establish the credibility of the model through any reasonable means. They argue that most model practitioners instinctively operate in this middle ground between objectivism and relativism in attempting to achieve model credibility. Whereas the HTA modellers interviewed recognised the importance of ensuring mutual understanding in the definition of the problem and structure of the model, none of the interviewees explicitly raised the issue of model credibility in considering the definitions of model error, although this may well have been assumed. The interviews demonstrate that HTA modellers tend to act in the manner described by Kleindorfer et al. , using all means at their disposal to demonstrate model validity and gain credibility, without necessarily referring to any philosophical underpinning for this mode of action.

Decision analytic modelling is essentially a Bayesian undertaking, both in the theoretical underpinnings and in the central role of the Bayes updating formula in the statistical analysis of decision problems. The Bayes formula provides a statistical mechanism for updating probabilities or beliefs in the light of new observations. There is a substantial methodological and applied literature on Bayesian methods. Most applications of the Bayesian approach in the HTA domain apply the updating procedure to refining estimates of model parameters within a model in the light of new data or in facilitating probabilistic sensitivity analysis. However, the same principle can be applied in approaching the validity of a model. In the Bayesian approach, the problem of validity is framed explicitly in terms of model credibility. Hence, the credibility, or measure of belief, in a theory or in our case mathematical model is updated in the light of new evidence. 38

Where m = model and d = data.

This latter approach is closely related to handling structural uncertainty in models through model averaging methods. A number of issues arise in this approach principally concerning the nature of the subjective prior probability of a theory or model, and the difficulty in accounting for the open nature of possible models, i.e. the impossibility of constructing a complete set of possible models to consider in the model averaging process or assigning an adequate assessment of the probability that none of the described models are correct. Furthermore, it should be noted that there is almost a complete absence of discussion of model verification or operational validity of the model within the decision-making process within the Bayesian literature.

Sargent39–41 in a series of closely related conference proceedings describes a refreshingly practical approach to verification and validity of models without recourse to epistemology or philosophy. Sargent uses a simple model of the model development process, reproduced in Figure 2, to develop different aspects of model validity and verification.

FIGURE 2.

Model of the modelling process. Reproduced with permission from the Institute of Electrical and Electronics Engineers, Sargent,41 ©2004 Institute of Electrical and Electronics Engineers, Inc.

In this scheme the overall validity of a modelling project comprises the validity of the conceptual model, the veracity of the implemented model to the conceptual model and the operational validity of the results and interpretation of the results in the decision-making process. It should be noted that using verification to describe the correct implementation of the conceptual model in the computerised model, has resonance with the logical empiricist philosophical approach to verification as discussed by Déry et al. 32 However, this highlights the fact that this definition of verification relies on their being an explicit and complete description of the conceptual model that acts as a specification of the computerised model. Where the description of the conceptual model is absent or incomplete, this separation between the concepts of verification and validation breaks down. As has been discussed in the previous section the absence of an adequate description of the conceptual model is a common feature of HTA modelling practice.

The methodological literature on model validation becomes sparse after the 1993 European Journal of Operational Research Special Issue,31 somewhat ironically given its stated aims of promoting discussion. Citations of the articles contained therein (searches undertaken January 2009) consist mainly of developments of validation approaches in specific application domains, for example Oreskes et al. 42 in the earth sciences and indeed this monograph in HTA. There are few generic methodological papers41,43,44 and these focus on pragmatic approaches to validation.

The HTA modellers interviewed demonstrate a pragmatic approach to model validation. Although there may certainly exist sound philosophical underpinnings for this stance, it is unclear, firstly, the extent to which this is a satisfactory position and secondly the extent to which practitioners are consciously implementing this as principled pragmatism or are implementing an unprincipled laissez faire approach to validation. In their seminal 1967 paper, Naylor et al. 28 described the problem of ascertaining the validity of a model as ‘perhaps the most elusive’ unresolved problem in computer simulation methods; the level of conflicting opinions and practices identified in the literature and in the interviews undertaken in this study indicates that there is still room for development in this area.

Introduction

This chapter presents a taxonomy of errors in HTA models, with the purpose of developing a common language for the development of methods and processes for identifying errors and avoiding errors in HTA models. The taxonomy provides a basis for relating current strategies for avoiding and identifying model errors to the types of errors they are intended to impact upon (see Explanatory analysis concerning methods for preventing errors in Chapters 6 and 7).

Identifying errors and avoiding errors imply two different aspects to the taxonomy. First, identifying errors implies a focus on error symptoms and error checking processes, whereas avoiding errors implies a focus on the process of model development and implementation. The taxonomy is not designed with the purpose of providing a checklist for the critical appraisal of models. As a result, where terminology such as ‘error of judgement’ is used, this is not meant to necessarily imply that a process is required or may exist for identifying a particular instance of judgment as being in error.

A two-stage process was used in the development of the taxonomy. First, an interview-based taxonomy was developed based upon the framework analysis of the in-depth interviews undertaken with a sample of the HTA modelling community. Before discussing individual error types, the interviewees were asked to describe their conception of the decision support process, through model development, implementation and use. This description of each individual’s perception of the model development process was then used as a vehicle for the discussion of the error types. This helped to ensure that the discussion of error types, and the balance of error types across the modelling process, is not biased towards different parts of the process by the nature of the interviews and perceptions of the interviewers. In addition, non-HTA literature regarding spreadsheet and programming error taxonomies was used as a basis for comment on the current understanding and as a means to further develop the taxonomy (see Review of taxonomy/classification literature).

Table 3 below presents the distribution of comments from the interviews across the major themes. It is notable that over 70% of comments related to the problem definition, structuring process and the use of evidence, whereas only 17% of comments related to actual errors in the implementation of models.

This focus among HTA modellers on errors in the ‘softer’ side of the model development process is supported by explicit comments throughout the interviews.

…my concerns about quality of outputs and accuracy is much more around the design of the conceptual model and asking the correct questions than it is about the implementing. Which is not, that they’re not both important, but that it’s easier to do guidelines around the technical aspects and it’s something that as modellers you’re more interested in, it’s your distributions and your methods and your environments, and it’s things that you can write down precisely and measure.

[I6]

But it’s much easier, I would think, to ask the wrong question and [that] not get found, and for that not to be picked up until the end, than it is to make a major mistake in coding and for that not to be picked up.

[I6]

These concerns relate directly to the synthesis of respondent’s views of what constitutes a model error (see Chapter 4), whereby inappropriate model assumptions and matters of judgement were clearly considered to represent model errors by some interviewees but not by others.

The next five sections on errors describe the interview-based taxonomy of model errors presented in Figure 3 structured according to the six major coded themes presented in Table 3, noting that the discussion of ‘error in operation of the model’ and ‘error in presentation and understanding of results’ have been merged, owing to the limited number of comments in these domains and their close association. The taxonomy is structured vertically according to the major coded themes that represent different aspects of the model development process constituting error domains, these major domains are each broken down into subdomains. Types of errors described by interviewees in each domain are then discussed; these are presented in column 2 of the taxonomy. For each type of error there has been an attempt to identify potential root causes of errors, these are presented in column 3 and are grouped together again in column 4 to identify major themes in the causes of error.

FIGURE 3.

Taxonomy of errors in HTA modelling: parts I–III.

Error in understanding the problem

Well, I guess it all stems from what is the decision problem, to start off with. So…you know, no matter what you’ve done, if you’ve got the decision problem wrong…basically it doesn’t matter how accurate the model is. You’re addressing effectively the wrong decision.

[I7]

The key error domains in the understanding and description of the decision problem identified by the interviewees were:

• error in the definition of population and subgroups

• error in the definition of interventions and comparators

• error in the definition of outcomes.

The charted interview data were examined to identify components that make up the description of a decision problem. Interviewees explicitly mentioned the definition of comparators, interventions and populations as being important in the description of the decision problem and potentially being subject to error. Five out of seven comments mentioned the comparators and/or interventions, which were sometimes indistinguishable, and three comments identified the population as a key element. Comments included:

Their conceptualisation missed out strategies that were highly likely to be the most cost-effective.

[I2]

…we have to be very clear from the outset that the specification of a decision problem…and I guess that’s got a number of different levels, which is, you know…what is the patient population that we’re interested in, what are the relevant potential subgroups that lie therein.

[I7]

Errors in the definition of populations, interventions and comparators seemed to be associated with subtle aspects of their definition, for example when considering populations it is important to appreciate the differences between subgroups, or in the definition of interventions or comparators it is the treatment strategy, that is the definition of treatment sequences, or missed comparators that give rise to errors.

It is also notable that none of the interviewees mentioned the definition of the outcome as potentially problematic when discussing the description of the problem. This probably reflects an assumption that has become axiomatic among HTA modellers that decision-makers require and are interested primarily in the generic cost-effectiveness outcome of the incremental cost per quality-adjusted life-year (QALY) gained. However, in discussing the presentation and understanding of results one interviewee indicated that presentation of disease-specific results was a key element in helping decision-makers to understand the nature of the model results:

So cost per QALY amalgamates a lot of things together and you might not really know what’s potentially driving the results quite clearly and you might not necessarily know that there is an implication of some of the things that you have done and you are not pulling them out so you are failing to really provide all the information to a decision-maker that they should have.

[I8]

It is noteworthy that interviewees focused solely on elements of the PICO (populations, interventions, comparators, outcomes) description of the decision problem. This may indicate that interviewees considered that the PICO definition adequately captures the typical HTA decision problem.

The errors identified by the interviewees applicable across these areas were:

• definitions incoherent with the clinical understanding of the disease

• inadequate description of the boundary and content of the decision problem.

These error types are illustrated in the interview responses presented below:

…but you can see that in some of the NICE reviews…where the question was asked incorrectly or the question was framed early on in a way that was not coherent with the clinicians’ understanding of the disease. And so the whole process flowed through and generated junk at the end.

[I6]

…they didn’t realise I’d have to build a treatment model alongside it and then they hadn’t invited the right people to come to it…[they] dropped the ball if they believed you could do a screening [model] without having a treatment model, but that wasn’t our error.

[I2]

It’s not an error in the model because we did what they asked us to, but they could have asked us a better question.

[I2]

The prime root cause suggested by the interviewees for errors in the description and understanding of the decision problem was a failure of communication between the stakeholders to the process. The interviewees identified three types of stakeholders:

• the decision-maker (NICE or client)

• the decision-modelling team (either an academic Assessment Group or an Outcomes Research organisation)

• clinical experts.

Error in the structure and methodology used

Three broad domains within model structuring can be distinguished within the interviews:

• development of a conceptual model of the disease including a description of its epidemiology, natural history and management

• selection of modelling methods

• moving from the conceptual model to an implemented model.

Error in the use of evidence

Discussions with interviewees concerning errors in the use of evidence focussed on:

• the use of evidence in informing the development of model structure

• the role of evidence in populating data inputs to the model

• generic evidence processes.

Error in implementation of the model

Errors in implementation identified by the interviewees can be classified into the following two domains:

• coding of individual cells

• coding of logic structures within the model.

Errors in operation of the model and model reporting

The key domains discussed by the interviewees in this section were:

• operation of the model

• presentation of results

• communication of results and conclusions.

Review of taxonomy/classification literature

Placing the interview taxonomy of errors in the context of literature on error classifications

Conclusion

The interviewees collectively demonstrated examples of all the major error types identified in the literature on errors in end-user developer spreadsheet systems. Taken together, the interview classification of modelling errors provides a basis for developing our understanding of errors and facilitating discussion of errors.

To take forward any strategy for reducing errors and improving the robustness of decision support, it is likely that further development of this classification of errors will be required. Particular attention should be shown in this regard when considering methods for improving the identification of errors.

The literature detailing classifications of spreadsheet errors, is based around the work of two key authors, Panko and Rajalingham. Six different versions of classifications have been identified, that all struggle in varying degrees to present classifications that are complete and mutually exclusive, whereas being rich enough to be useful. Ko and Myers48 identified explicitly the often complex chains or networks of errors and faults that can lead to a failure and related these to research on the cognitive background to human error. The principle of complex chains underpinning errors is described in the interviews but the relation of modelling errors to human cognitive processes is poorly understood. This layer of complexity is poorly developed in the spreadsheet error literature and not explicitly discussed in the HTA modelling interviews.

The error category termed by Panko ‘context errors’ hides a multitude of different error causes. For example this category contains errors associated with breakdowns in domain/disease knowledge, modelling skills/knowledge, programming skills/knowledge, mathematical logic and statistical methods knowledge. All of these represent areas of judgement where the identification of and definition of error are problematic and may include at their most extreme, violations of methods and processes. This area requires specific attention when moving towards strategies to reduce errors in HTA modelling.

Overview

This chapter explores the procedures and techniques for preventing the introduction of errors in health economic models discussed by the interview respondents. As with the other themes that emerged from interrogation of the interview data, a descriptive account of discussions surrounding the avoidance of model errors was developed through the categorisation and classification of the charted transcript data. The interview data were analysed in terms of the methods and procedures for error avoidance as well as the experiences, views and perceived importance of these methods. Within the qualitative analysis, five classifications emerged from the analysis:

1. Mutual understanding, which relates to joint understanding and communication between stakeholders to the processes of model development and use. This includes decision-makers and clients, clinical advisors and other stakeholders, health economic modellers and other members of the research team such as systematic reviewers and information specialists.

2. Model complexity and its perceived relationship to the generation of errors.

3. Housekeeping processes, which relate to features and techniques that can be incorporated into implemented models to alert the modeller to possible errors and ensure the analyst is aware of the current state of the model.

4. Guidelines and interviewees’ perceptions of their necessary content, role and limitations.

5. Skills and training, which explores the skills, abilities and knowledge that modellers working within the HTA process should possess.

The techniques and processes identified within the descriptive analysis were related to the model taxonomy to examine gaps between types of model error and current approaches currently adopted by the HTA community to ensure their avoidance (see Table 7).

Mutual understanding

Model complexity

One key issue that modellers are regularly faced with when designing or developing a model concerns its complexity. The interview data suggested that modellers do not want to spend a substantial amount of time designing and implementing a highly complex model when a simpler design would have sufficed. Nor, however, do they want to simplify the implementation of the model to such an extent that clinically relevant factors are lost or omitted. It should be noted that the concept of model complexity is in itself subjective (what is considered a complex model by one individual may be simplistic to another).

• model complexity and the importance of transparency

• arguments in favour of model simplicity

• arguments in favour of model complexity.

Housekeeping

Housekeeping concerns features and techniques employed within an implementation model (and its development) to facilitate error checking. Four approaches were discussed by respondents:

• use of automatic checks and flags

• use of model settings which are visible on the screen

• standard layout conventions

• naming of cells and cell ranges.

Guidelines

Eleven of twelve respondents discussed the potential role of modelling guidelines. However, the term ‘guidelines’ may have been interpreted differently; at one extreme some may view guidelines as a standardised model development map that prescribes good practice under any eventuality, whereas others may have more relaxed views of what constitutes a guideline, for example, reference cases, critical appraisal checklists and good practice suggestions.

Skills and training

The final subtheme relating to the prevention of errors within models concerned the skills and training of the modeller and the broader research team. Interview discussions covered a range of skills; the qualitative data were interrogated with respect to four classifications:

• skills related to the design and development of models

• skills related to locating errors in models

• skills in statistical analysis techniques

• background and other abilities of the modeller.

Explanatory analysis concerning methods for preventing errors

Table 7 presents an explanatory analysis which attempts to link the current methods for preventing model errors to the taxonomy of model error presented earlier (see Chapter 5). The leftmost column details ‘methods’ currently used to avoid error raised by the interview respondents. These are loosely defined as either processes or techniques; the former relate to issues in the model development process, whereas the latter relate to techniques of implementation. These methods have been mapped against the types of model error they are likely (or intended) to avoid, based on the interview data and through detailed discussion among the authors. As far as possible these approaches for avoiding errors have been mapped to the root cause of the model error; however, in several instances the primary source or cause of error was unclear. The mapping presented in Table 7 may not exhaustively capture every type of error that these methods address. In addition, given the variation in the elicited model development processes of individual respondents (see Chapter 3), the use of the full range of methods for avoiding errors is not necessarily typical.

A number of emergent issues arose from the interpretation of the mapping presented in Table 7. A simple crosstabulation of the number of avoidance strategies for each error domain gives an indication of the focus of the modelling community on different areas of model validity. Clearly, the range of methods and approaches to avoiding model errors is considerably broader than the housekeeping techniques focused on avoiding errors in the implementation of the model; only one-fifth of the methods target the implementation and operation of the model. The remaining four-fifths target the conceptual validity of the model. Very little emphasis is placed on presenting and communicating results and conclusions. An emergent issue that is evident in the synthesis concerns the nature, and balance, of processes and techniques for avoiding model errors. In particular, the techniques detailed are explicit, and can be interpreted as relating to how something should be done, for example, implementing a specific model layout. On the other hand, the processes for avoiding errors recognise that there is a need to be addressed and acknowledge that something should be done as part of model development; however, in many cases this is not accompanied by a clear strategy for achieving the required goal. This issue particularly affects those aspects relating to the definition of the decision problem and conceptual modelling. A number of requirements are identified to achieve clarity and mutual understanding; these are expressed as process requirements (e.g. establishment of long-term clinical input) to avoid errors in the description of the decision problem. Whereas a number of techniques are suggested by the interviewees, for example, sketching out clinical pathways, these are not framed within an overall strategy for structuring complex problems.

Placing the findings of the interviews in the context of the literature on avoiding errors

Fostering mutual understanding between the modelling team and all stakeholders to the decision-making process is targeted at achieving model credibility. The issue of defining appropriate level of complexity is well discussed within the literature; however, no objective methods are suggested for dealing with this issue. Literature on problem-structuring methods suggests that the focus should be on developing adequate models, that are ‘owned’ by the decision-maker. This in turn, highlights the importance of transparency of reporting, implementation and interpretation of models.

Overview

This chapter presents a qualitative analysis of discussions concerning procedures and techniques that are employed to identify whether, where and why errors have been introduced into models. As part of the Framework analysis, a number of classifications emerged from the qualitative data.

• Check face validity with clinicians.

• Do the results appear reasonable?

• Test model behaviour.

• Is the model able to reproduce its inputs?

• Can the model replicate other data not used in its construction?

• Compare answers with the answers generated by alternative models.

• Peer review of models.

• Double checking of input values.

• Double-programming.

Check face validity with experts

A commonly cited approach used in the identification of model errors concerned checking face validity of interim and final model results with people who know about the disease and treatment(s) under consideration. This has relevance both in the avoidance and the identification of errors; in this case the primary differences in its role concerning whether the error is found before or after the decision has been made, or whether a model is being scrutinised by clinical experts acting on behalf of the decision-maker (e.g. reviewing models submitted by manufacturers within the Single Technology Appraisal process). Key discussions surrounding this issue have therefore been presented (see Chapter 6).

Do the results appear reasonable?

A common issue concerning the identification of model errors involved whether the results appeared to be ‘reasonable’. As noted earlier (see Chapter 4), the ‘reasonableness’ of model results, and whether they match preconceived expectations, were raised as dimensions of what constitutes a model error.

…there’s a level of validation which can happen at [the] end of any model, because, you know, you want to make sure it’s actually giving you numbers which aren’t ridiculous…

[I7]

…I suppose, that there’s a kind of issue of judgement in terms of whether it looks, whether the model itself looks reasonable…

[I9]

The ability of a modeller to develop an expectation of the answer has been discussed previously (see Chapter 6, Skills and training). As highlighted through these previous discussions, the source of these expectations differed between respondents. For example, one respondent suggested that an expectation of the answer can be formed by the modeller through an examination of the key features of the disease and the evidence used to inform the model.

…you’ll look at the key things which…are going on, but you want this right at the beginning of the process. We call it the sniff test which is whether the result smells right…

[I6]

In practice, this approach echoes the idea of ‘skeleton models’; ‘back-of-the-envelope’ models, which adopt a high level of abstraction to generate an expectation of the model result. One respondent highlighted how this might work in practice; however, as already noted (see Chapter 4), although a model result may match one’s expectations, this does not provide a guarantee that the model is error-free.

…my back of the envelope said this was going to be about forty grand per QALY and there were some subgroups that might be interesting. If you come to the end, or you come to a draft model of the results, and you have something radically different from that, then there should be a reason. If it comes out and it’s four grand per QALY then it just smells wrong, and something has changed between the decision problem you thought you were looking at in the beginning and the data that’s actually been implemented…

[I6]

…do the results, if it [the model] says the ICER is £50,000…does that seem about right using the back of the envelope sort of techniques…you’re never going to be able to confirm that exactly without the full model full calculations but you should be able to at least justify [it] or not within a fairly…certain range…

[I11]

The same respondent described the action which he would take if there were a significant difference between the expectation and the answer from the model:

That’s the sort of detective phase I think, identifying which bit is different, so your fag packet might have got differences in costs divided by differences in QALYs…so firstly which of those two or both is different from what the model is generating, and if it’s say the difference in costs, is there a cost of intervention or the cost of the comparator, which one’s wrong?

[I11]

Similarly, another respondent stated that he estimates what the model output should be using pen and paper at the start of the project; if this approximated result is not within a reasonable tolerance of the results from the final model, he then attempts to establish which is wrong and why.

…I regularly work out the answer before I do the project and see if I am right or not, and if I don’t end up with the answer I think I have, work out why I was mistaken in the beginning, why my original calculation was wrong or why the new calculation is wrong…

[I2]

That’s why it’s interesting, when it doesn’t match your back of the fag packet answer, you’ve got to be damn sure you are right…

[I2]

A different respondent stated that when a model is operating in an unexpected fashion then one of the approaches is to build up the same relationships and logic and try to track down which part of the model is responsible for the error.

…if issues are identified in the model during construction or in quality control and it’s one of those really annoying, why the hell is this doing this? Then one of the approaches is, and again this is experience and not written down. It’s out the model and I’m going to go down to back of the fag packet, start again and just build up the same thing, in essence, not replicate it, but just build up the same kind of relationship and logic to see whether that’s at all possible or try and track down which part of that process must be going wrong to hit those kind of numbers…

[I10]

The same respondent suggested that such approaches should be a standard part of the model development process.

All the good modellers do and the bad ones don’t…

[I10]

Testing model behaviour

Testing model behaviour relates to the process of changing the input values in a way which will result in a change in the results, and examining whether the change was expected (note there is a direct link to the methods for developing expectations described previously). One respondent suggested a series of logical tests involving the efficacy of interventions as a means of identifying potential errors within a model.

…if I increase the efficacy of an intervention, the effectiveness ought to go up and the cost-effectiveness ought to go down. If I switch the inputs for the two arms of the model, I should get the opposite result out of it at the end. If I put the same inputs into both arms, I should get a null…

[I6]

…as you start to think through the model you start to think of an expectation of what the results are going to be and given a certain change how you think the results should change and the more specific you get in terms of the structure of a model and the data inputs the more specific you can be about how you expect the change to be so if it isn’t in the direction or the magnitude that you are expecting, you are expecting that you have probably made a mistake somewhere…

[I8]

Another respondent states that it is possible to prospectively define a set of input parameters and develop an expectation of their impact on the model output.

…well…one thing you can do is you’d define a set of input[s], and you know what will be the set of output[s]. And with that, you know exactly…whether the results are right or wrong…

[I1]

…you see testing as a black box where you define input and you know what output you are expecting. So you change the inputs, and you see how the outputs change…

[I1]

The same respondent highlighted that techniques exist that can facilitate this process of design and analysis; however these are not currently used within HTA modelling.

Imagine you have five inputs that can have different values, each of them, and you know what the outputs should be for the values. What would be the minimum number of combinations that you can do with your five inputs in order to get to be sure that the outputs are correct…so there are techniques that tell you how to define your testing inputs, then, in order to get the outputs…

[I1]

Another suggestion concerned the use of model functionality tests such as altering assumptions concerning the initial distribution of patients in the model.

…model functionality tests…what happens if you set the starting population to be all in one state rather than another…

[I11]

Another approach was to test extreme input values in terms of stability and robustness of model outputs.

…taking extreme settings as inputs as ways of checking that the model is resilient to use its input…

[I11]

Whereas many of the respondents discussed the use of model testing to identify errors, it was suggested that the use of such techniques did not guarantee that errors would be identified (a similar point was made concerned developing expectations; see Do the results appear reasonable?). One respondent suggested that the model scenario may match previously determined expectations as a result of luck rather than as a consequence of internal consistency within the model.

…if you can test this by a kind of univariate type sensitive analysis approach where you’re saying I think this is going to have a huge impact or at least if I increase this value, the result, you know, that value should increase rather than decrease…you may have the right direction, seeing the right magnitude of change, but actually purely by luck…It’s just pure luck that there happens to be the case and there’s some error in there.

[I9]

Other examples of model tests included setting all utility parameters to zero, adjusting transition probabilities and assessing the impact, and setting the parameters in each arm of the model to the same value and checking that the results were identical.

Simple things like if you set utility values in the model to zero is the output of the utility zero in each arm? If you set the data points in each arm equal do you get an ICER of zero or not…

[I4]

…when you increase one of the transitions which you expect to reduce the utility then the utility goes downward. So you play around with it and make sure its pretty much behaving like you want…

[I4]

Although the majority of interviewees discussed the techniques they adopted, one respondent adopted a more normative stance, suggesting that the testing model behaviour should be a standard part of the model development process.

…every time you change something, you should as a matter of course just check that the ICER goes in the right direction…A new parameter, a parameter changes or a new variable or a new state or any of these things get added you rerun the model, check the ICER, did it go in the right direction…

[I2]

…what we have at the moment is a model prior to final or full analysis being run that we are happy with and at that point it is only really when we are starting the final stages of the model when you are producing results, some of the results, especially those exploring extreme values, that your problems become fully apparent and then the question becomes do we need to refine the model or is our model giving reasonably good predictions or any biases that we have within our model reasonable over the plausible range of those values and only becoming uncertain over implausible values. So we are actually maybe trying to specify at what points our model is going to fall over…

[I8]

…you’d hope that that sort of iterative process, the sort of extreme values would kind of dig up the major ones…

[I7]

Is the model able to reproduce its inputs?

Testing internal validity, that is, checking that the model can generate the inputs used to populate it, was also discussed by participants. This corresponds to Eddy’s second order of model validation. 59 Several respondents suggested that they would typically use this approach as part of the model checking process; examples included testing incidence data and survival analysis used to inform model parameters.

…I suppose it does depend on you know if you have taken for example, national statistics, cancer incidence data to go into your model then you very much expect the incidence coming out of it…

[I11]

…we might do some kind of level of validation in terms of…model survival curves [do they] look anything like the data itself? And so there would be some kind of, where it’s possible, internal sort of validation on data inputs, and I think that would be done on a sort of statistical basis…

[I7]

However, some participants’ views of the utility of this method were negative, indicating that it may be a necessary but not sufficient condition of any model.

…internal validity means that you get out what you put in, big deal, fine…you have to do that, but it doesn’t get you anywhere. It doesn’t tell you that your model is right. It only means that what you put in – if you put rubbish in it, then you’ll get rubbish out of it. So all it, all it means is that, you know, you’ve got the mathematics…right…

[I3]

Another respondent highlighted that there may be instances whereby one would not expect the model outputs to match the input data used to inform it; however, this respondent did express a preference for demonstrating internal consistency where possible.

…even if you have good reasons your numbers won’t come out exactly the same and some of those might be overriding reasons, because of the way you are trying to use the trial for a particular model in a particular setting. If it’s possible to build a model in such a way that you can show consistency with a clinical trial then I think you should do that. That is a useful check…

[I12]

Interestingly, one respondent suggested that testing the internal consistency of the model did not necessarily take place within the model itself, but may be undertaken as part of a premodel analysis. The perceived benefit of this approach was that it may ensure that the data have been interpreted correctly before their incorporation into the model.

…I would just make sure that my survival models and other data that I have pulled from the trial is making sense, quite separately from any economic decision model. Can I reproduce these things in a way that looks right? Just to make sure I have done the analysis correctly and understood the results…

[I12]

Can the model replicate other data not used in its construction?

An additional aspect of model validation discussed by interview participants concerned the ability of the model to replicate data that have not been used in its construction; this corresponds to Eddy’s third order of model validation. 59 One respondent stated that this level of model validation is rarely possible given issues of data scarcity and the general principle of using all relevant evidence to inform the model.

What does that mean – external validity? Well, if it means accurately predicting something from a different source, well, that’s all right, except that most of the modelling that we do, that I’ve ever done, is obliged to use every source that’s available! Because of data scarcity. So basically there you know, you cannot do external validation…

[I3]

The same respondent highlighted other problems concerning the relationship between the data used to populate the model and its relationship to evidence collected in the future.

…when you’re on a chronic model, by the time you get 10 years into the future, the whole circumstances have changed, anyway. The epidemiology’s changed, the interventions have changed, the population’s changed, you know, everything’s gone…if somebody predicted it and put it in a sealed envelope, it’s bound to be wrong. Does that mean that the model was wrong? No, because it was talking on the basis of what was known at the time and the circumstances at the time…no, predictive validation is impossible anyway. So the whole notion of model validation is either trivial or impossible…

[I3]

One respondent highlighted that even if such analyses were possible, time constraints may represent a barrier for this level of validation.

…we talked about [whether] our model had predicted the trial results but by that point you are on another project so you’ve got too many things going round your head and we’ve never actually done it, and then it’s probably too late to say you know that model three years ago, it’s wrong…

[I2]

Compare answers with the answers generated by alternative models

Interviewees also discussed comparing the de novo model results against results from previously published studies. However, this is likely to be problematic as a method of validation because models which attempt to address similar decision problems may have employed different structures, assumptions and parameters. As noted by one respondent, it may be necessary to accept that some difference in results is to be expected; however, the underlying cause of the difference may not be clear.

…it’s a judgement, but you’re not expecting to get absolutely the same result, but, you know, the convergent validity, let’s call it that…the differences between these models, do they, come down to fundamental differences in structure, differences in conceptualisation, differences in scope, or is it the parameter inputs? And what would obviously be reassuring in that situation is change the parameter inputs to suit those of the existing models produced, you know, similar results…

[I9]

…there are two levels I guess of validity, one of which is does…your model…look kind of reasonable in terms of what other people have done…

[I7]

One respondent suggested that this aspect of model checking should run alongside model development; however, there may be a danger of becoming reliant on the structure and assumptions of previous models (see Chapter 3, Discussion of model development process). Indeed, the presence of differences between models does not guarantee that either model is ‘wrong’, rather such differences may be explained by the modellers’ preference for a given set of assumptions and methods.

…if somebody else has got a model of a different answer to it, we wouldn’t automatically say it’s an error. Because, you know, we’ve got our own preferred set of assumptions and ways of dealing with it…

[I7]

Nonetheless, it was suggested by one respondent that certain aspects of the model should always follow accepted conventions and that failure to do so limits the comparability of model results.

So the accepted convention for discounting is that you discount after the first year in whole numbers…so this 12 months is not discounted, the next is discounted at 1 year and 2 years and so on…well, you know, people will correctly use the correct discounting formula from point zero, so they are discounting every day from then onwards…it’s not technically incorrect, but it is conventionally incorrect and inappropriate if you are then going to make a comparison with another study in the same area that’s discounted in a different way. So then your answers are then not comparable…

[I3]

Peer review of models

The majority of respondents discussed internal peer review as a common element of the model development process representing a useful means of identifying errors. Perhaps the principal reason for peer review was that ‘your own mistakes are the hardest to see’ [I4]. For example, one respondent stated that completed models are passed to a senior member of staff who is charged with the task of ‘breaking’ the model.

We have a process here where we have a sort of a peer review, so we have enough experienced staff that we could give the the draft model to one of the other project leaders and ask them to spend a few days kicking it about to see if there are errors or mistakes in it…

[I6]

Often, but not always, this was suggested to involve a modeller who has not been involved in the design or implementation of the model; however, some respondents suggested that meetings were held and model results presented throughout the development process (for example, in the supervision of junior staff).

…you might need someone external to the modelling to find those kind of errors…

[I1]

A key issue in the internal model peer review was that the reviewer must be capable of understanding the complexities of the model.

…you would like to think that the people checking it are reputable and able so they will understand the complexities that they have put into the model whereas others might just dismiss them…

[I12]

Similarly, one respondent suggested that external scrutiny of the model may also be seen as an ideal; however, this may be constrained by time and concerns regarding intellectual property and a lack of willingness to share ideas. As a consequence, it was suggested that internal peer review is likely to be a viable second-best alternative:

…if there were time, you would want to send it out to some independent expert who would go through it all with a fine-tooth comb and tell you everything that’s wrong with it. In practice that doesn’t happen, it doesn’t happen because there isn’t the time to do it, and secondly, it doesn’t happen, because if you’re doing anything that’s genuinely original, you know, you don’t want to run the risk of anybody else picking up all your ideas. So, you know, there are problems on that front. The best that you can do is to try and expose it to somebody probably internal, who hasn’t been involved in the development, and let them go through it and say, ‘Well, can you find any obvious flaws here?’ And that’s probably the best that, in reality, we can usually do…

[I3]

It was suggested that the benefits of peer review did not lie solely in terms of identifying errors, but also in consolidating the modeller’s own understanding of the decision problem and the justification for the modelling approach.

…there‘s also a stage of exposing the model at that stage to I suppose to people within the health economics group but also outside of that, so I think there is a, well, I think there’s a benefit in terms of, again, just going over the way the model’s been put together…

[I9]

Several respondents highlighted the importance of model scrutiny through peer review because of the model’s political importance, as well as a feeling that if the peer review failed to identify substantial errors the modeller could be more confident that the model was error-free.

We have had people check my…model because that’s a huge, politically important piece of work…

[I2]

…these things are subject to severe scrutiny and if they find something tiny well they’ve looked through a lot…it was a pure transcription error, I put the wrong number in and it was picked up by the technical lead…the thought was if that had been checked with that much scrutiny and that was all they could find then there is actually a good chance that the rest of it was correct…

[I5]

However, despite the potential benefits of model peer review, some respondents raised concerns about the sensitivity of peer review (before publication) in detecting model errors:

…I remain baffled as to how general reviewers assess the quality of models without looking at them…but how a journal can perform a meaningful review on the basis of a three and a half thousand word manuscript describing a highly technical model…

[I6]

We try to publish in journals with the highest citation rating. Those are almost inevitably clinical journals. Clinical journals use clinicians and generally speaking the reviews for clinical journals are the older clinicians who’ve had very little direct exposure to economic evaluation, modelling or anything. They are pure clinicians. They don’t know what they are reviewing…

[I3]

Double checking of input values

One commonly cited process for identifying model errors involved double-checking model input parameters in terms of its incorporation within the model and cross-checking against the original sources from which parameter value(s) were derived, e.g. clinical trial publications. In relation to definitions of model error, these processes primarily concern ensuring that data have been transcribed and linked within the model as the modeller intended.

…we ask them to look at the accuracy of the input data, so that’s been incorporated into the model appropriately…

[I6]

…there is a lot of checking to make sure the data is right not just so you talked about what the meaning of the data points are but also make sure that the actual values are what they should be so we step through the model and check, go through each data point and make sure it’s right…

[I4]

One respondent suggested that the effectiveness of such checking activities may be constrained by time availability and the tedium of the task.

…I mean, checking and double-checking, really. The only way really is having the time and the discipline to go through, crosscheck everything you do several times…

[I3]

One respondent stipulated that such checks must involve checking confidence intervals as well as mean input values.

It wasn’t the point estimate that had been entered wrong it was the confidence interval and things like that are difficult to spot…

[I12]

Double-programming

The issue of ‘double-programming’ was discussed implicitly or explicitly as a potential approach to identifying model errors. For the purposes of this analysis, double-programming relates to the development of the same mathematical model in two different platforms; a distinction should therefore be drawn between double-programming and the development of ‘skeleton’ models discussed earlier (see Chapter 3). Although respondents generally agreed the double-programming was a valuable activity, there was variation surrounding the extent to which double-programming was undertaken as a standard component of the model development process.

Explanatory analysis concerning methods for preventing errors

Table 8 presents an explanatory synthesis which attempts to link the current procedures and techniques for identifying model errors to the taxonomy of model error presented in Chapter 6. The layout and interpretation of the table, and the methods of synthesis used in its construction, are essentially the same as that for Table 7.

Placing the findings of the interviews in the context of the literature on identifying model error

The literature revealed a number of validation techniques which have been developed in the field of computer science. Many of the techniques described in the literature were discussed by the interviewees. These included: comparison of the model with previous modelling of the same situation, seeking face validity through asking clinicians who have specialised in the disease process or medical condition being investigated to comment on the model, retrospective and prospective predictive validation, degenerate testing in which the value of a single parameter is changed and the behaviour of the model is examined to determine if it is in line with the expected behaviour, and extreme condition testing in which the model parameters are assigned extreme values and the behaviour of the model is investigated. Another process that is described in the literature and the interviews is the process of assigning the model fixed values that will facilitate manual calculation of the results or interim results, analogous to the pen and paper approach.

A number of processes were listed that were either not discussed or discussed in different terms. These include animation, operational graphics and trace testing. In animation, for example the number of patients in each health state across time might be displayed graphically across time to determine whether the behaviour of the model is in line with the expected behaviour. An example of trace testing might involve tracking the proportion of patients in each health state numerically across time; the exhibited behaviour can then be compared to determine agreement with the expected behaviour.

The literature also describes Turing testing, in which the ability of an expert to distinguish between observed phenomena and output from a simulation model is used as a test of the model’s validity. This process was not described by the interviewees and may have some potential for applications in HTA modelling.

Given the overlap between the identification and the prevention of model errors, the processes and techniques relating to clinical involvement in model development and model scrutiny are not repeated here. One key point is immediately noticeable from the interpretation of Table 8; many of the methods used to identify errors in the model involve specific techniques (rather than processes) that may be applied once the model is complete, in progress or under scrutiny by a third-party. However, the specific target of the techniques, i.e. the types of error that the technique is intended to identify is not always clear. Indeed, the majority of methods do not appear to map directly to specific error types in the taxonomy, but rather they may be used to identify symptoms of errors, the true source of which may relate to any point in the taxonomy of model errors. For example, the identification of a mismatch between actual model results and the derivation of prior expectations from a previous model may be suggestive of the presence of an error within the model, yet its root cause may be entirely unclear (there may even be legitimate reasons for such differences). This represents a considerable challenge in the peer review of models. The same may be true of certain black-box validation techniques; only the methods which test the underlying logic of the model can guarantee the presence of an error. In this sense, the methods outlined in Table 8 that can be mapped directly to specific aspects of the taxonomy (e.g. switching model inputs between arms) are diagnostic in nature; mismatches in model results and expectations are indicative of the presence of model error. Conversely, those techniques and procedures which map to any or all points in the taxonomy are effectively non-specific model screening methods.

Barriers

Barriers to model checking activities discussed by interviewees are summarised in Box 1. The interviewees commonly focused on resource constraints (time constraints, people, money) and organisational/process constraints. Two further barriers included the perceived importance and priority of model checking activities and modellers’ confidence in current model checking procedures.

Views concerning the nature of current barriers to model checking varied between respondents. Two respondents highlighted a trade-off between time available for developing the model and time available for checking the model:

…if you spend all of your time doing quality control you don’t have as much to actually build a model.

[I6]

Time was highlighted as a barrier to some of the more aspirational model checking activities such as double-programming, particularly for more complex models. Two respondents also suggested a relationship between time constraints and model validity issues in terms of the use of an appropriate methodology and plausible assumptions.

I’ve gone back and gone well why did I make that assumption? Actually a better assumption was this, but I’ve gone with an assumption because it was easier, quicker and they need the report tomorrow.

[I2]

A second respondent also suggested that while one may wish to explore whether more sophisticated models would have made a difference, this is constrained by time. Human resource constraints were also raised by several respondents, including availability of modellers as well as other stakeholders within the model development process, e.g. clinical experts providing input on clinical practice.

You can have clinician involvement in terms of the clinician who is removed from the client and can offer an independent voice, not always possible due to time constraints, it really needs to be done quite quickly to organise the necessary meetings, preparatory literature and so on…

[I12]

Alongside physical human resources, the dimensions of human constraints also included the availability of sufficient levels of expertise in scrutinising HTA models. Financial constraints were noted explicitly by some respondents, and implicitly by others. However, even with sufficient financial resources, it was suggested that more extensive model checking activities may not identify and avoid all model errors potentially because of the tedious nature of model checking activities and a lack of confidence therein.

…we don’t get paid enough, the cheaper option is to get someone else to go through it but after they’ve been through a thousand lines of coding and not spotted an error, do they give up and not check the next thousand.

[I4]

…it is a matter of painstakingly going through and seeing ‘yes, this number and this model and where does this correspond to in the report’, but that is not something you know – it is not something that is not guaranteed to work just because it is a tedious process…you are not going to have time to go through and find every single number.

[I5]

The above statements serve to highlight some of the gaps between the breadth of error types presented in the taxonomy and the effectiveness of current model checking activities to identify and avoid these (see Tables 7 and 8). Other barriers included developing models ‘too quickly’ and organisational barriers such as working within groups in which a shared understanding is not maintained. One respondent raised a number of less tangible barriers to model checking in which the onus is placed on the modeller rather than the constraints of the decision-making process. Barriers included a lack of proper time management, a lack of appreciation of the need to check models to identify errors, failure to adopt housekeeping approaches and a lack of accountability for model errors.

To explore the priority allocated to model checking activities interviewees were asked ‘if you had to increase checking which other activities would you drop?’ and/or ‘where in the development process would additional monies/time be allocated?’. Some of the comments indicated that a low priority was attached to these activities, indicating that simple resource constraints were not necessarily the key barrier.

I don’t think we’d drop anything. I’d spend less time on validation…well, most of these projects are all underfunded anyway…so, we’d probably again apportion that money into those other activities, and again…put less on validation.

[I7]

…if we are under severe time pressure we try to cut back on what the outputs delivered will be, so if we were hoping to do a value of information analysis we will chop that, if we were hoping to do sophisticated sensitivity analysis we would chop that, just drop blocks of work rather than compromising the quality of key work which is the fundamental outputs in the model.

[I4]

Facilitators

Discussion of potential facilitators to model checking was typically brief, with few respondents suggesting how any of the above barriers could be ameliorated. Interviews suggested that setting up internal processes to ensure that things are not done in a hurry, that projects are adequately staffed and that work is divided up appropriately might facilitate improved error checking activities. In contrast to the barriers to model checking, one respondent from an Outcomes Research organisation suggested that in his experience clients would provide a budget for quality control; this view was not reflected in any respondents working within an academic setting. One interviewee suggested the use of a time delay within the model development process to allow improved internal review:

If we had a 3-month delay in every project where you handed it in…and then you came back in 3 months to read your report and re-examine your assumptions, your logic and what you had written in the report, your report would be a hell of a lot better, which is why the peer review.

[I2]

Strengths and weaknesses of the study

The structure for considering the validity of qualitative research described by Mays and Pope24 is used here to examine the strengths and weaknesses of this study. Mays and Pope identify the following key issues of validity: triangulation of methods, respondent validation, transparency of methods, reflexivity and fair dealing.

This study uses two primary sources of evidence, interviews with HTA modellers and reviews of the literature, this twin approach has the strength of providing a triangulation of different research methods, helping to ensure a complete coverage of issues and allowing a convergent validity of the results and conclusions to be examined.

Respondent validation was undertaken with the HTA modellers interviewed to ensure that the extracts from the transcripts used within the analysis were taken within context and did not misrepresent the views of the interviewees. This validation was undertaken by sending the interviewees a near final draft of the report, including the complete interview analyses and principle conclusions. The communication invited both general comment on the content of the report and specific comment on the above validity issues. All interviewees were happy that their views had not been misrepresented.

To ensure clarity in reporting of methods the report has been subjected to internal peer review from a qualitative researcher, mixed methodologist, modellers not involved in the interviews and a systematic reviewer.

Attention was paid in the research design to ensure as far as possible that the research methods and particular perspectives of the researchers did not bias the results. Three individuals were involved in the interviews to avoid bias associated with having a single interviewer. Ten of the twelve interviews were undertaken in pairs, with one lead interviewer and one interviewer responsible for ensuring complete coverage of the questioning and capturing of loose ends raised during the interview. To ensure congruence between the lead interviewers the interview pilots and the first interview were undertaken by the two lead interviewers together. Some of the interviewees were known personally by the interviewers, this may have affected the nature of some of the responses. These effects may have been positive in aiding an open communication during the interviews, and negative in leading to interviewers assuming the nature of some of the interview responses.

Regarding the definition of errors, the preliminary questioning about the full breadth of the modelling process may have biased interviewees towards a definition that encompassed all stages of the process. An alternative method of questioning would have been to commence discussion on technical errors of implementation and let interviewees explore the definition outward until they defined their own boundary. However, this approach may equally have led to an artificially constrained definition of errors.

In the analysis the importance ascribed to individual accounts is only partially related to their frequency. The importance ascribed to comments is at least potentially subject to bias arising from the prior beliefs and prejudices of the analysts, this was guarded against by attention to deviant cases and by undertaking elements of the analyses in teams.

The interview sample was designed to ensure fair representation of organisational perspectives in the analysis. After the first draft of the descriptive interview analysis, the authors checked the distribution of comments used in the report across the different interviewees. Although there was some variation in the number of comments ascribed to different individuals, all interviewees were represented in the analysis. No rebalancing of the report was therefore deemed necessary.

With regard to the review of literature on error classifications, one potential weakness was that the literature focused on spreadsheet and program development rather than specifically on model development. It is assumed that this evidence is transferable to the modelling domain. It should also be noted that the identified literature concerning programming errors was very narrow despite broad searches being undertaken.

The interviews focused on the process of modelling for HTA decision-making; however, much of this discussion focuses on generic modelling issues and one strength of this work is the potential validity outside its immediate domain. Perhaps the main impact of the focus on HTA modelling is in the area of understanding the decision problem. The HTA modellers interviewed moved directly to a discussion of the PICO definition of a problem scope. When moving to different decision-making areas, this process of defining the problem would have to be more broadly based with an initial focus on defining the key characteristics of the decision problem. Thereafter, much of the modelling discussion appears to be generic.

Current understanding of modelling errors in the HTA community

There is a general consensus that an important part of the definition of what constitutes an error is its impact on decision-making. This indicates that a pragmatic philosophical approach is generally taken across the HTA modelling community. Due to the nature of the interview process it cannot be assumed that the absence of an explicit discussion of such a theoretical underpinning means that individuals do not use such a framework in informing their modelling activities.

Although the interview discussions all implied this common pragmatic outlook, there was no common language used in the discussion of modelling errors and furthermore there was a somewhat inconsistent approach to the boundaries of what constitutes an error. For instance, when asked explicitly about the definition of model error, there was a tendency for participants to exclude matters of judgement from being errors and focus on what have been termed slips and lapses. However, discussion of slips and lapses comprised less than 20% of the discussion on types of errors. When considering how individual elements of the modelling process might contribute to flaws in decision-making, the interviewees devoted 70% of the discussion to softer elements of the process of defining the decision question and conceptual modelling, mostly the realms of judgement, skills, experience and training.

The original focus of this research, and for consistency the terminology throughout this report, has been in terms of errors in models. However, in the light of the previous discussion of error boundaries and when considering methods of improving modelling practice and processes and examining skills and training requirements, it may be more useful to refer to modelling risks rather than the more black and white term modelling errors.

During the interviews, a number of respondents discussed the concepts of validation and verification. Where validation and verification were discussed, there was a consistency among the participants in their interpretation. Verification was taken to mean the process of ensuring that the computer model correctly implemented the intended model, that is the implementation of the model is free of errors. Validation meant the process of ensuring that a model is fit for purpose, some interviewees explicitly noting that this concept might subsume verification. There was some discussion that related to the concept of credibility; however, this was not developed in any formal sense. As a consequence, issues of validation and verification are central to the discussion of errors. Sargent41 provides a definition of overall validity, encapsulated within Figure 2, as being comprised of conceptual model validity, verification of the computer model, and operational validity of the use of the model in addressing the real world problem. The definitions of verification and validation provided by the interviewees are compatible with these definitions. These definitions are therefore recommended as the basis for further discussion of these topics.

The qualitative analysis highlights considerable variation in modelling practice across the HTA modelling community. This may be in part explained by the apparent variations in expertise, background and experience between the respondents. One particular area of variation concerned the explicit demonstration of conceptual modelling.

Although there was consensus surrounding the definition of verification, to make this a measurable and useful concept there needs to be an explicit and complete description (or design specification) of the intended or conceptual model. A common theme in the modelling process discussion was the absence of a fully transparent description of the intended model (indeed for some interviewees this barely exists at all). So even the distinction between verification and validation breaks down in any practically useful sense.

The methodological literature on verification and validation of models makes reference to the Hermeneutic philosophical position that recognises that objectivity is unachievable and suggests that meaning is created through intersubjective communication. This literature supports the proposition that it is the interaction between the modeller and client in developing mutual understanding of a model that establishes a model’s significance and its warranty. 35 This position highlights that model credibility is the central concern to decision-makers in using models as an input to the decision-making process. This highlights the point that the concept of model validation should not be externalised from the decision-makers and the decision-making process.

A taxonomy of HTA modelling errors

The interviewees collectively demonstrated examples of all the major error types identified in the literature on errors in end-user developer spreadsheet systems. Taken together, the interview classification of modelling errors provides a basis for developing our understanding of errors and facilitating discussion of errors. The literature detailing classifications of spreadsheet errors is based around the work of two key authors, Panko and Rajalingham. Six different versions of classifications have been identified, each of which struggle to present classifications that are complete and mutually exclusive while being sufficiently rich to be useful, the HTA error classifications similarly struggle with these issues.

Ko explicitly identifies complex chains or networks of errors and faults that can lead to a failure. This layer of complexity is not explicitly discussed either in the spreadsheet error literature or in the HTA modelling interviews. Further development of model errors and risks should reflect this complexity in the relationship between errors in models and failures in decision-making. Existing research on the cognitive basis of human error should be brought into the examination of modelling errors.

To take forward any strategy for reducing errors and improving the robustness of decision support, it is likely that further development of this classification of errors will be required, specifically with regard to considering methods for improving the retrospective identification of errors. This is because the taxonomy does not seek to classify errors by symptoms.

The error category termed by Panko as ‘context errors’ hides a multitude of different error causes. For example, this category contains errors associated with breakdowns in domain/disease knowledge, modelling skills/knowledge, programming skills/knowledge, mathematical logic and statistical methods knowledge. All of these represent areas of judgement where the identification and definition of error are problematic and may include at their most extreme, violations of methods and processes. This area requires specific attention when moving towards strategies to avoid errors in HTA modelling.

Current strategies for avoiding errors

The qualitative analysis suggests that a range of techniques and procedures are currently used to avoid errors in HTA models. Importantly, there is some degree of overlap between methods used to identify errors and methods used to avoid errors in models; the distinction between avoidance and identification methods is determined by the point at which the strategy is applied. These strategies for error avoidance are loosely defined as either processes or techniques; the former relate to issues in the model development process, whereas the latter relate to techniques of implementation. Generally, the techniques are explicit and can be interpreted as relating to how something should be done, for example, implementing a specific model layout. Conversely, the processes recognise an unfulfilled requirement and acknowledge that something should be done as part of model development, yet in many cases this is not accompanied by a clear strategy for achieving the required goal. Current methods for avoiding errors include engaging with clinical experts, engaging with the client or decision-maker to ensure mutual understanding, developing written documentation of the proposed model, explicit conceptual modelling, e.g. using diagrams and sketches, stepping through skeleton models with experts, ensuring transparency in reporting, adopting standard housekeeping techniques, and ensuring that those parties involved in the model development process have sufficient training in relevant disciplines (e.g. health economics, statistics, systematic reviewing), as well as ensuring appropriate skills in model programming, model scrutiny and general problem structuring. Evidently, the range of strategies and approaches for avoiding model errors is considerably broader than housekeeping techniques alone; only one-fifth of the strategies target the implementation and operation of the model. The remaining four-fifths target the conceptual validity of the model. Very little emphasis is placed on how model results are conveyed to users.

Clarity and mutual understanding, specifically in the definition of the decision problem and conceptual modelling, are identified as key issues. Currently adopted strategies supporting these aspects of model development are expressed as process requirements, e.g. establishment of long-term clinical input and iterative negotiation with the decision-maker or client may be used to avoid errors in the description of the decision problem. Although a number of techniques were suggested by the interviewees, for example, sketching out clinical pathways, their use appears to be partial, and the extent of their use appears to vary considerably between individual modellers. Hence, while there is an acknowledgement of the importance of these methods, their current implementation is not framed within an overall strategy for structuring complex problems.

Current strategies for identifying errors in HTA models

Current methods for identifying errors in HTA models include checking face validity, assessing whether model results appear reasonable, black-box testing strategies, testing internal consistency and predictive validity, checking model input values, double-programming and peer review. These strategies largely relate to specific techniques (rather than processes) that may be applied by third-party scrutiny. However, the specific target of the techniques, i.e. the types of error that the technique is intended to identify, is not always clear. Indeed, the majority of methods do not appear to map directly to specific error types in the taxonomy, but rather they may be used to identify symptoms of errors, the true source of which may relate to any point in the taxonomy of model errors. For example, the identification of a mismatch between actual model results and the derivation of prior expectations from a previous model may be suggestive of the presence of an error, yet its root cause may be entirely unclear. This represents a considerable challenge in the peer review of models. The same may be true of certain black-box validation techniques; only the tests of the underlying logic of the model guarantee the presence of an error. In this sense, the methods outlined in Table 8 which do map directly to specific aspects of the taxonomy are diagnostic in nature; mismatches in model results and expectations are indicative of the presence of model error. Those aspects which map to any or all points in the taxonomy are effectively non-specific model screening methods; the presence of differences between models and prior expectations are not necessarily the result of a model error.

Recommendations

• Published definitions of overall model validity comprising conceptual model validation, verification of the computer model, and operational validity of the use of the model in addressing the real-world problem are consistent with the views expressed by the HTA community and are therefore recommended as the basis for further discussions of model credibility.

• Discussions of model credibility should focus on risks, including errors of implementation, errors in matters of judgement and violations – violations being defined as puffery, fraud or breakdowns in operational procedures.

• Discussions of modelling risks should reflect the potentially complex network of cognitive breakdowns that lead to errors in models and subsequent failures in decision support. Existing research concerning the cognitive basis of human error should be brought into the examination of modelling errors.

• There is a need to develop a better understanding of the skills requirements for the development, operation and use of HTA models.

• The qualitative interviews highlighted a number of barriers to model checking. However, it was indicated that increasing time and resources would not necessarily improve model checking activities without a matched increase in their prioritisation.

• The authors take the view, as supported within the methods literature, that it is the interaction between the modeller and client in developing mutual understanding of a model that establishes a model’s significance and its warranty. This position highlights that model credibility is the central concern to decision-makers in using models. It is crucial then that the concept of model validation should not be externalised from the decision-makers and the decision-making process.

Research recommendations

Contributions of authors

Jim Chilcott led the project. Jim Chilcott, Paul Tappenden and Suzy Paisley were responsible for the design of the study. Jim Chilcott, Paul Tappenden and Andrew Rawdin undertook all in-depth interviews. All authors contributed towards the qualitative synthesis methods and implementation. Diana Papaioannou carried out the literature searches.

Disclaimers

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

Health Technology Assessment reports published to date

1. Home parenteral nutrition: a systematic review.

   By Richards DM, Deeks JJ, Sheldon TA, Shaffer JL.

2. Diagnosis, management and screening of early localised prostate cancer.

   A review by Selley S, Donovan J, Faulkner A, Coast J, Gillatt D.

3. The diagnosis, management, treatment and costs of prostate cancer in England and Wales.

   A review by Chamberlain J, Melia J, Moss S, Brown J.

4. Screening for fragile X syndrome.

   A review by Murray J, Cuckle H, Taylor G, Hewison J.

5. A review of near patient testing in primary care.

   By Hobbs FDR, Delaney BC, Fitzmaurice DA, Wilson S, Hyde CJ, Thorpe GH, et al.

6. Systematic review of outpatient services for chronic pain control.

   By McQuay HJ, Moore RA, Eccleston C, Morley S, de C Williams AC.

7. Neonatal screening for inborn errors of metabolism: cost, yield and outcome.

   A review by Pollitt RJ, Green A, McCabe CJ, Booth A, Cooper NJ, Leonard JV, et al.

8. Preschool vision screening.

   A review by Snowdon SK, Stewart-Brown SL.

9. Implications of socio-cultural contexts for the ethics of clinical trials.

   A review by Ashcroft RE, Chadwick DW, Clark SRL, Edwards RHT, Frith L, Hutton JL.

10. A critical review of the role of neonatal hearing screening in the detection of congenital hearing impairment.

    By Davis A, Bamford J, Wilson I, Ramkalawan T, Forshaw M, Wright S.

11. Newborn screening for inborn errors of metabolism: a systematic review.

    By Seymour CA, Thomason MJ, Chalmers RA, Addison GM, Bain MD, Cockburn F, et al.

12. Routine preoperative testing: a systematic review of the evidence.

    By Munro J, Booth A, Nicholl J.

13. Systematic review of the effectiveness of laxatives in the elderly.

    By Petticrew M, Watt I, Sheldon T.

14. When and how to assess fast-changing technologies: a comparative study of medical applications of four generic technologies.

    A review by Mowatt G, Bower DJ, Brebner JA, Cairns JA, Grant AM, McKee L.

1. Antenatal screening for Down’s syndrome.

   A review by Wald NJ, Kennard A, Hackshaw A, McGuire A.

2. Screening for ovarian cancer: a systematic review.

   By Bell R, Petticrew M, Luengo S, Sheldon TA.

3. Consensus development methods, and their use in clinical guideline development.

   A review by Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CFB, Askham J, et al.

4. A cost–utility analysis of interferon beta for multiple sclerosis.

   By Parkin D, McNamee P, Jacoby A, Miller P, Thomas S, Bates D.

5. Effectiveness and efficiency of methods of dialysis therapy for end-stage renal disease: systematic reviews.

   By MacLeod A, Grant A, Donaldson C, Khan I, Campbell M, Daly C, et al.

6. Effectiveness of hip prostheses in primary total hip replacement: a critical review of evidence and an economic model.

   By Faulkner A, Kennedy LG, Baxter K, Donovan J, Wilkinson M, Bevan G.

7. Antimicrobial prophylaxis in colorectal surgery: a systematic review of randomised controlled trials.

   By Song F, Glenny AM.

8. Bone marrow and peripheral blood stem cell transplantation for malignancy.

   A review by Johnson PWM, Simnett SJ, Sweetenham JW, Morgan GJ, Stewart LA.

9. Screening for speech and language delay: a systematic review of the literature.

   By Law J, Boyle J, Harris F, Harkness A, Nye C.

10. Resource allocation for chronic stable angina: a systematic review of effectiveness, costs and cost-effectiveness of alternative interventions.

    By Sculpher MJ, Petticrew M, Kelland JL, Elliott RA, Holdright DR, Buxton MJ.

11. Detection, adherence and control of hypertension for the prevention of stroke: a systematic review.

    By Ebrahim S.

12. Postoperative analgesia and vomiting, with special reference to day-case surgery: a systematic review.

    By McQuay HJ, Moore RA.

13. Choosing between randomised and nonrandomised studies: a systematic review.

    By Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C.

14. Evaluating patient-based outcome measures for use in clinical trials.

    A review by Fitzpatrick R, Davey C, Buxton MJ, Jones DR.

15. Ethical issues in the design and conduct of randomised controlled trials.

    A review by Edwards SJL, Lilford RJ, Braunholtz DA, Jackson JC, Hewison J, Thornton J.

16. Qualitative research methods in health technology assessment: a review of the literature.

    By Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P.

17. The costs and benefits of paramedic skills in pre-hospital trauma care.

    By Nicholl J, Hughes S, Dixon S, Turner J, Yates D.

18. Systematic review of endoscopic ultrasound in gastro-oesophageal cancer.

    By Harris KM, Kelly S, Berry E, Hutton J, Roderick P, Cullingworth J, et al.

19. Systematic reviews of trials and other studies.

    By Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F.

20. Primary total hip replacement surgery: a systematic review of outcomes and modelling of cost-effectiveness associated with different prostheses.

    A review by Fitzpatrick R, Shortall E, Sculpher M, Murray D, Morris R, Lodge M, et al.

1. Informed decision making: an annotated bibliography and systematic review.

   By Bekker H, Thornton JG, Airey CM, Connelly JB, Hewison J, Robinson MB, et al.

2. Handling uncertainty when performing economic evaluation of healthcare interventions.

   A review by Briggs AH, Gray AM.

3. The role of expectancies in the placebo effect and their use in the delivery of health care: a systematic review.

   By Crow R, Gage H, Hampson S, Hart J, Kimber A, Thomas H.

4. A randomised controlled trial of different approaches to universal antenatal HIV testing: uptake and acceptability. Annex: Antenatal HIV testing – assessment of a routine voluntary approach.

   By Simpson WM, Johnstone FD, Boyd FM, Goldberg DJ, Hart GJ, Gormley SM, et al.

5. Methods for evaluating area-wide and organisation-based interventions in health and health care: a systematic review.

   By Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney PGJ.

6. Assessing the costs of healthcare technologies in clinical trials.

   A review by Johnston K, Buxton MJ, Jones DR, Fitzpatrick R.

7. Cooperatives and their primary care emergency centres: organisation and impact.

   By Hallam L, Henthorne K.

8. Screening for cystic fibrosis.

   A review by Murray J, Cuckle H, Taylor G, Littlewood J, Hewison J.

9. A review of the use of health status measures in economic evaluation.

   By Brazier J, Deverill M, Green C, Harper R, Booth A.

10. Methods for the analysis of quality-of-life and survival data in health technology assessment.

    A review by Billingham LJ, Abrams KR, Jones DR.

11. Antenatal and neonatal haemoglobinopathy screening in the UK: review and economic analysis.

    By Zeuner D, Ades AE, Karnon J, Brown J, Dezateux C, Anionwu EN.

12. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses.

    A review by Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al.

13. ‘Early warning systems’ for identifying new healthcare technologies.

    By Robert G, Stevens A, Gabbay J.

14. A systematic review of the role of human papillomavirus testing within a cervical screening programme.

    By Cuzick J, Sasieni P, Davies P, Adams J, Normand C, Frater A, et al.

15. Near patient testing in diabetes clinics: appraising the costs and outcomes.

    By Grieve R, Beech R, Vincent J, Mazurkiewicz J.

16. Positron emission tomography: establishing priorities for health technology assessment.

    A review by Robert G, Milne R.

17. The debridement of chronic wounds: a systematic review.

    By Bradley M, Cullum N, Sheldon T.

18. Systematic reviews of wound care management: (2) Dressings and topical agents used in the healing of chronic wounds.

    By Bradley M, Cullum N, Nelson EA, Petticrew M, Sheldon T, Torgerson D.

19. A systematic literature review of spiral and electron beam computed tomography: with particular reference to clinical applications in hepatic lesions, pulmonary embolus and coronary artery disease.

    By Berry E, Kelly S, Hutton J, Harris KM, Roderick P, Boyce JC, et al.

20. What role for statins? A review and economic model.

    By Ebrahim S, Davey Smith G, McCabe C, Payne N, Pickin M, Sheldon TA, et al.

21. Factors that limit the quality, number and progress of randomised controlled trials.

    A review by Prescott RJ, Counsell CE, Gillespie WJ, Grant AM, Russell IT, Kiauka S, et al.

22. Antimicrobial prophylaxis in total hip replacement: a systematic review.

    By Glenny AM, Song F.

23. Health promoting schools and health promotion in schools: two systematic reviews.

    By Lister-Sharp D, Chapman S, Stewart-Brown S, Sowden A.

24. Economic evaluation of a primary care-based education programme for patients with osteoarthritis of the knee.

    A review by Lord J, Victor C, Littlejohns P, Ross FM, Axford JS.

1. The estimation of marginal time preference in a UK-wide sample (TEMPUS) project.

   A review by Cairns JA, van der Pol MM.

2. Geriatric rehabilitation following fractures in older people: a systematic review.

   By Cameron I, Crotty M, Currie C, Finnegan T, Gillespie L, Gillespie W, et al.

3. Screening for sickle cell disease and thalassaemia: a systematic review with supplementary research.

   By Davies SC, Cronin E, Gill M, Greengross P, Hickman M, Normand C.

4. Community provision of hearing aids and related audiology services.

   A review by Reeves DJ, Alborz A, Hickson FS, Bamford JM.

5. False-negative results in screening programmes: systematic review of impact and implications.

   By Petticrew MP, Sowden AJ, Lister-Sharp D, Wright K.

6. Costs and benefits of community postnatal support workers: a randomised controlled trial.

   By Morrell CJ, Spiby H, Stewart P, Walters S, Morgan A.

7. Implantable contraceptives (subdermal implants and hormonally impregnated intrauterine systems) versus other forms of reversible contraceptives: two systematic reviews to assess relative effectiveness, acceptability, tolerability and cost-effectiveness.

   By French RS, Cowan FM, Mansour DJA, Morris S, Procter T, Hughes D, et al.

8. An introduction to statistical methods for health technology assessment.

   A review by White SJ, Ashby D, Brown PJ.

9. Disease-modifying drugs for multiple sclerosis: a rapid and systematic review.

   By Clegg A, Bryant J, Milne R.

10. Publication and related biases.

    A review by Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ.

11. Cost and outcome implications of the organisation of vascular services.

    By Michaels J, Brazier J, Palfreyman S, Shackley P, Slack R.

12. Monitoring blood glucose control in diabetes mellitus: a systematic review.

    By Coster S, Gulliford MC, Seed PT, Powrie JK, Swaminathan R.

13. The effectiveness of domiciliary health visiting: a systematic review of international studies and a selective review of the British literature.

    By Elkan R, Kendrick D, Hewitt M, Robinson JJA, Tolley K, Blair M, et al.

14. The determinants of screening uptake and interventions for increasing uptake: a systematic review.

    By Jepson R, Clegg A, Forbes C, Lewis R, Sowden A, Kleijnen J.

15. The effectiveness and cost-effectiveness of prophylactic removal of wisdom teeth.

    A rapid review by Song F, O’Meara S, Wilson P, Golder S, Kleijnen J.

16. Ultrasound screening in pregnancy: a systematic review of the clinical effectiveness, cost-effectiveness and women’s views.

    By Bricker L, Garcia J, Henderson J, Mugford M, Neilson J, Roberts T, et al.

17. A rapid and systematic review of the effectiveness and cost-effectiveness of the taxanes used in the treatment of advanced breast and ovarian cancer.

    By Lister-Sharp D, McDonagh MS, Khan KS, Kleijnen J.

18. Liquid-based cytology in cervical screening: a rapid and systematic review.

    By Payne N, Chilcott J, McGoogan E.

19. Randomised controlled trial of non-directive counselling, cognitive–behaviour therapy and usual general practitioner care in the management of depression as well as mixed anxiety and depression in primary care.

    By King M, Sibbald B, Ward E, Bower P, Lloyd M, Gabbay M, et al.

20. Routine referral for radiography of patients presenting with low back pain: is patients’ outcome influenced by GPs’ referral for plain radiography?

    By Kerry S, Hilton S, Patel S, Dundas D, Rink E, Lord J.

21. Systematic reviews of wound care management: (3) antimicrobial agents for chronic wounds; (4) diabetic foot ulceration.

    By O’Meara S, Cullum N, Majid M, Sheldon T.

22. Using routine data to complement and enhance the results of randomised controlled trials.

    By Lewsey JD, Leyland AH, Murray GD, Boddy FA.

23. Coronary artery stents in the treatment of ischaemic heart disease: a rapid and systematic review.

    By Meads C, Cummins C, Jolly K, Stevens A, Burls A, Hyde C.

24. Outcome measures for adult critical care: a systematic review.

    By Hayes JA, Black NA, Jenkinson C, Young JD, Rowan KM, Daly K, et al.

25. A systematic review to evaluate the effectiveness of interventions to promote the initiation of breastfeeding.

    By Fairbank L, O’Meara S, Renfrew MJ, Woolridge M, Sowden AJ, Lister-Sharp D.

26. Implantable cardioverter defibrillators: arrhythmias. A rapid and systematic review.

    By Parkes J, Bryant J, Milne R.

27. Treatments for fatigue in multiple sclerosis: a rapid and systematic review.

    By Brañas P, Jordan R, Fry-Smith A, Burls A, Hyde C.

28. Early asthma prophylaxis, natural history, skeletal development and economy (EASE): a pilot randomised controlled trial.

    By Baxter-Jones ADG, Helms PJ, Russell G, Grant A, Ross S, Cairns JA, et al.

29. Screening for hypercholesterolaemia versus case finding for familial hypercholesterolaemia: a systematic review and cost-effectiveness analysis.

    By Marks D, Wonderling D, Thorogood M, Lambert H, Humphries SE, Neil HAW.

30. A rapid and systematic review of the clinical effectiveness and cost-effectiveness of glycoprotein IIb/IIIa antagonists in the medical management of unstable angina.

    By McDonagh MS, Bachmann LM, Golder S, Kleijnen J, ter Riet G.

31. A randomised controlled trial of prehospital intravenous fluid replacement therapy in serious trauma.

    By Turner J, Nicholl J, Webber L, Cox H, Dixon S, Yates D.

32. Intrathecal pumps for giving opioids in chronic pain: a systematic review.

    By Williams JE, Louw G, Towlerton G.

33. Combination therapy (interferon alfa and ribavirin) in the treatment of chronic hepatitis C: a rapid and systematic review.

    By Shepherd J, Waugh N, Hewitson P.

34. A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies.

    By MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AMS.

35. Intravascular ultrasound-guided interventions in coronary artery disease: a systematic literature review, with decision-analytic modelling, of outcomes and cost-effectiveness.

    By Berry E, Kelly S, Hutton J, Lindsay HSJ, Blaxill JM, Evans JA, et al.

36. A randomised controlled trial to evaluate the effectiveness and cost-effectiveness of counselling patients with chronic depression.

    By Simpson S, Corney R, Fitzgerald P, Beecham J.

37. Systematic review of treatments for atopic eczema.

    By Hoare C, Li Wan Po A, Williams H.

38. Bayesian methods in health technology assessment: a review.

    By Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR.

39. The management of dyspepsia: a systematic review.

    By Delaney B, Moayyedi P, Deeks J, Innes M, Soo S, Barton P, et al.

40. A systematic review of treatments for severe psoriasis.

    By Griffiths CEM, Clark CM, Chalmers RJG, Li Wan Po A, Williams HC.

1. Clinical and cost-effectiveness of donepezil, rivastigmine and galantamine for Alzheimer’s disease: a rapid and systematic review.

   By Clegg A, Bryant J, Nicholson T, McIntyre L, De Broe S, Gerard K, et al.

2. The clinical effectiveness and cost-effectiveness of riluzole for motor neurone disease: a rapid and systematic review.

   By Stewart A, Sandercock J, Bryan S, Hyde C, Barton PM, Fry-Smith A, et al.

3. Equity and the economic evaluation of healthcare.

   By Sassi F, Archard L, Le Grand J.

4. Quality-of-life measures in chronic diseases of childhood.

   By Eiser C, Morse R.

5. Eliciting public preferences for healthcare: a systematic review of techniques.

   By Ryan M, Scott DA, Reeves C, Bate A, van Teijlingen ER, Russell EM, et al.

6. General health status measures for people with cognitive impairment: learning disability and acquired brain injury.

   By Riemsma RP, Forbes CA, Glanville JM, Eastwood AJ, Kleijnen J.

7. An assessment of screening strategies for fragile X syndrome in the UK.

   By Pembrey ME, Barnicoat AJ, Carmichael B, Bobrow M, Turner G.

8. Issues in methodological research: perspectives from researchers and commissioners.

   By Lilford RJ, Richardson A, Stevens A, Fitzpatrick R, Edwards S, Rock F, et al.

9. Systematic reviews of wound care management: (5) beds; (6) compression; (7) laser therapy, therapeutic ultrasound, electrotherapy and electromagnetic therapy.

   By Cullum N, Nelson EA, Flemming K, Sheldon T.

10. Effects of educational and psychosocial interventions for adolescents with diabetes mellitus: a systematic review.

    By Hampson SE, Skinner TC, Hart J, Storey L, Gage H, Foxcroft D, et al.

11. Effectiveness of autologous chondrocyte transplantation for hyaline cartilage defects in knees: a rapid and systematic review.

    By Jobanputra P, Parry D, Fry-Smith A, Burls A.

12. Statistical assessment of the learning curves of health technologies.

    By Ramsay CR, Grant AM, Wallace SA, Garthwaite PH, Monk AF, Russell IT.

13. The effectiveness and cost-effectiveness of temozolomide for the treatment of recurrent malignant glioma: a rapid and systematic review.

    By Dinnes J, Cave C, Huang S, Major K, Milne R.

14. A rapid and systematic review of the clinical effectiveness and cost-effectiveness of debriding agents in treating surgical wounds healing by secondary intention.

    By Lewis R, Whiting P, ter Riet G, O’Meara S, Glanville J.

15. Home treatment for mental health problems: a systematic review.

    By Burns T, Knapp M, Catty J, Healey A, Henderson J, Watt H, et al.

16. How to develop cost-conscious guidelines.

    By Eccles M, Mason J.

17. The role of specialist nurses in multiple sclerosis: a rapid and systematic review.

    By De Broe S, Christopher F, Waugh N.

18. A rapid and systematic review of the clinical effectiveness and cost-effectiveness of orlistat in the management of obesity.

    By O’Meara S, Riemsma R, Shirran L, Mather L, ter Riet G.

19. The clinical effectiveness and cost-effectiveness of pioglitazone for type 2 diabetes mellitus: a rapid and systematic review.

    By Chilcott J, Wight J, Lloyd Jones M, Tappenden P.

20. Extended scope of nursing practice: a multicentre randomised controlled trial of appropriately trained nurses and preregistration house officers in preoperative assessment in elective general surgery.

    By Kinley H, Czoski-Murray C, George S, McCabe C, Primrose J, Reilly C, et al.

21. Systematic reviews of the effectiveness of day care for people with severe mental disorders: (1) Acute day hospital versus admission; (2) Vocational rehabilitation; (3) Day hospital versus outpatient care.

    By Marshall M, Crowther R, Almaraz- Serrano A, Creed F, Sledge W, Kluiter H, et al.

22. The measurement and monitoring of surgical adverse events.

    By Bruce J, Russell EM, Mollison J, Krukowski ZH.

23. Action research: a systematic review and guidance for assessment.

    By Waterman H, Tillen D, Dickson R, de Koning K.

24. A rapid and systematic review of the clinical effectiveness and cost-effectiveness of gemcitabine for the treatment of pancreatic cancer.

    By Ward S, Morris E, Bansback N, Calvert N, Crellin A, Forman D, et al.

25. A rapid and systematic review of the evidence for the clinical effectiveness and cost-effectiveness of irinotecan, oxaliplatin and raltitrexed for the treatment of advanced colorectal cancer.

    By Lloyd Jones M, Hummel S, Bansback N, Orr B, Seymour M.

26. Comparison of the effectiveness of inhaler devices in asthma and chronic obstructive airways disease: a systematic review of the literature.

    By Brocklebank D, Ram F, Wright J, Barry P, Cates C, Davies L, et al.

27. The cost-effectiveness of magnetic resonance imaging for investigation of the knee joint.

    By Bryan S, Weatherburn G, Bungay H, Hatrick C, Salas C, Parry D, et al.

28. A rapid and systematic review of the clinical effectiveness and cost-effectiveness of topotecan for ovarian cancer.

    By Forbes C, Shirran L, Bagnall A-M, Duffy S, ter Riet G.

29. Superseded by a report published in a later volume.

30. The role of radiography in primary care patients with low back pain of at least 6 weeks duration: a randomised (unblinded) controlled trial.

    By Kendrick D, Fielding K, Bentley E, Miller P, Kerslake R, Pringle M.

31. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients.

    By McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, et al.

32. A rapid and systematic review of the clinical effectiveness and cost-effectiveness of paclitaxel, docetaxel, gemcitabine and vinorelbine in non-small-cell lung cancer.

    By Clegg A, Scott DA, Sidhu M, Hewitson P, Waugh N.

33. Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives.

    By Brookes ST, Whitley E, Peters TJ, Mulheran PA, Egger M, Davey Smith G.

34. Depot antipsychotic medication in the treatment of patients with schizophrenia: (1) Meta-review; (2) Patient and nurse attitudes.

    By David AS, Adams C.

35. A systematic review of controlled trials of the effectiveness and cost-effectiveness of brief psychological treatments for depression.

    By Churchill R, Hunot V, Corney R, Knapp M, McGuire H, Tylee A, et al.

36. Cost analysis of child health surveillance.

    By Sanderson D, Wright D, Acton C, Duree D.

1. A study of the methods used to select review criteria for clinical audit.

   By Hearnshaw H, Harker R, Cheater F, Baker R, Grimshaw G.

2. Fludarabine as second-line therapy for B cell chronic lymphocytic leukaemia: a technology assessment.

   By Hyde C, Wake B, Bryan S, Barton P, Fry-Smith A, Davenport C, et al.

3. Rituximab as third-line treatment for refractory or recurrent Stage III or IV follicular non-Hodgkin’s lymphoma: a systematic review and economic evaluation.

   By Wake B, Hyde C, Bryan S, Barton P, Song F, Fry-Smith A, et al.

4. A systematic review of discharge arrangements for older people.

   By Parker SG, Peet SM, McPherson A, Cannaby AM, Baker R, Wilson A, et al.

5. The clinical effectiveness and cost-effectiveness of inhaler devices used in the routine management of chronic asthma in older children: a systematic review and economic evaluation.

   By Peters J, Stevenson M, Beverley C, Lim J, Smith S.

6. The clinical effectiveness and cost-effectiveness of sibutramine in the management of obesity: a technology assessment.

   By O’Meara S, Riemsma R, Shirran L, Mather L, ter Riet G.

7. The cost-effectiveness of magnetic resonance angiography for carotid artery stenosis and peripheral vascular disease: a systematic review.

   By Berry E, Kelly S, Westwood ME, Davies LM, Gough MJ, Bamford JM, et al.

8. Promoting physical activity in South Asian Muslim women through ‘exercise on prescription’.

   By Carroll B, Ali N, Azam N.

9. Zanamivir for the treatment of influenza in adults: a systematic review and economic evaluation.

   By Burls A, Clark W, Stewart T, Preston C, Bryan S, Jefferson T, et al.

10. A review of the natural history and epidemiology of multiple sclerosis: implications for resource allocation and health economic models.

    By Richards RG, Sampson FC, Beard SM, Tappenden P.

11. Screening for gestational diabetes: a systematic review and economic evaluation.

    By Scott DA, Loveman E, McIntyre L, Waugh N.

12. The clinical effectiveness and cost-effectiveness of surgery for people with morbid obesity: a systematic review and economic evaluation.

    By Clegg AJ, Colquitt J, Sidhu MK, Royle P, Loveman E, Walker A.

13. The clinical effectiveness of trastuzumab for breast cancer: a systematic review.

    By Lewis R, Bagnall A-M, Forbes C, Shirran E, Duffy S, Kleijnen J, et al.

14. The clinical effectiveness and cost-effectiveness of vinorelbine for breast cancer: a systematic review and economic evaluation.

    By Lewis R, Bagnall A-M, King S, Woolacott N, Forbes C, Shirran L, et al.

15. A systematic review of the effectiveness and cost-effectiveness of metal-on-metal hip resurfacing arthroplasty for treatment of hip disease.

    By Vale L, Wyness L, McCormack K, McKenzie L, Brazzelli M, Stearns SC.

16. The clinical effectiveness and cost-effectiveness of bupropion and nicotine replacement therapy for smoking cessation: a systematic review and economic evaluation.

    By Woolacott NF, Jones L, Forbes CA, Mather LC, Sowden AJ, Song FJ, et al.

17. A systematic review of effectiveness and economic evaluation of new drug treatments for juvenile idiopathic arthritis: etanercept.

    By Cummins C, Connock M, Fry-Smith A, Burls A.

18. Clinical effectiveness and cost-effectiveness of growth hormone in children: a systematic review and economic evaluation.

    By Bryant J, Cave C, Mihaylova B, Chase D, McIntyre L, Gerard K, et al.

19. Clinical effectiveness and cost-effectiveness of growth hormone in adults in relation to impact on quality of life: a systematic review and economic evaluation.

    By Bryant J, Loveman E, Chase D, Mihaylova B, Cave C, Gerard K, et al.

20. Clinical medication review by a pharmacist of patients on repeat prescriptions in general practice: a randomised controlled trial.

    By Zermansky AG, Petty DR, Raynor DK, Lowe CJ, Freementle N, Vail A.

21. The effectiveness of infliximab and etanercept for the treatment of rheumatoid arthritis: a systematic review and economic evaluation.

    By Jobanputra P, Barton P, Bryan S, Burls A.

22. A systematic review and economic evaluation of computerised cognitive behaviour therapy for depression and anxiety.

    By Kaltenthaler E, Shackley P, Stevens K, Beverley C, Parry G, Chilcott J.

23. A systematic review and economic evaluation of pegylated liposomal doxorubicin hydrochloride for ovarian cancer.

    By Forbes C, Wilby J, Richardson G, Sculpher M, Mather L, Riemsma R.

24. A systematic review of the effectiveness of interventions based on a stages-of-change approach to promote individual behaviour change.

    By Riemsma RP, Pattenden J, Bridle C, Sowden AJ, Mather L, Watt IS, et al.

25. A systematic review update of the clinical effectiveness and cost-effectiveness of glycoprotein IIb/IIIa antagonists.

    By Robinson M, Ginnelly L, Sculpher M, Jones L, Riemsma R, Palmer S, et al.

26. A systematic review of the effectiveness, cost-effectiveness and barriers to implementation of thrombolytic and neuroprotective therapy for acute ischaemic stroke in the NHS.

    By Sandercock P, Berge E, Dennis M, Forbes J, Hand P, Kwan J, et al.

27. A randomised controlled crossover trial of nurse practitioner versus doctor-led outpatient care in a bronchiectasis clinic.

    By Caine N, Sharples LD, Hollingworth W, French J, Keogan M, Exley A, et al.

28. Clinical effectiveness and cost – consequences of selective serotonin reuptake inhibitors in the treatment of sex offenders.

    By Adi Y, Ashcroft D, Browne K, Beech A, Fry-Smith A, Hyde C.

29. Treatment of established osteoporosis: a systematic review and cost–utility analysis.

    By Kanis JA, Brazier JE, Stevenson M, Calvert NW, Lloyd Jones M.

30. Which anaesthetic agents are cost-effective in day surgery? Literature review, national survey of practice and randomised controlled trial.

    By Elliott RA Payne K, Moore JK, Davies LM, Harper NJN, St Leger AS, et al.

31. Screening for hepatitis C among injecting drug users and in genitourinary medicine clinics: systematic reviews of effectiveness, modelling study and national survey of current practice.

    By Stein K, Dalziel K, Walker A, McIntyre L, Jenkins B, Horne J, et al.

32. The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature.

    By Crow R, Gage H, Hampson S, Hart J, Kimber A, Storey L, et al.

33. The effectiveness and cost-effectiveness of imatinib in chronic myeloid leukaemia: a systematic review.

    By Garside R, Round A, Dalziel K, Stein K, Royle R.

34. A comparative study of hypertonic saline, daily and alternate-day rhDNase in children with cystic fibrosis.

    By Suri R, Wallis C, Bush A, Thompson S, Normand C, Flather M, et al.

35. A systematic review of the costs and effectiveness of different models of paediatric home care.

    By Parker G, Bhakta P, Lovett CA, Paisley S, Olsen R, Turner D, et al.

1. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.

   By Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J.

2. Systematic review of the effectiveness and cost-effectiveness, and economic evaluation, of home versus hospital or satellite unit haemodialysis for people with end-stage renal failure.

   By Mowatt G, Vale L, Perez J, Wyness L, Fraser C, MacLeod A, et al.

3. Systematic review and economic evaluation of the effectiveness of infliximab for the treatment of Crohn’s disease.

   By Clark W, Raftery J, Barton P, Song F, Fry-Smith A, Burls A.

4. A review of the clinical effectiveness and cost-effectiveness of routine anti-D prophylaxis for pregnant women who are rhesus negative.

   By Chilcott J, Lloyd Jones M, Wight J, Forman K, Wray J, Beverley C, et al.

5. Systematic review and evaluation of the use of tumour markers in paediatric oncology: Ewing’s sarcoma and neuroblastoma.

   By Riley RD, Burchill SA, Abrams KR, Heney D, Lambert PC, Jones DR, et al.

6. The cost-effectiveness of screening for Helicobacter pylori to reduce mortality and morbidity from gastric cancer and peptic ulcer disease: a discrete-event simulation model.

   By Roderick P, Davies R, Raftery J, Crabbe D, Pearce R, Bhandari P, et al.

7. The clinical effectiveness and cost-effectiveness of routine dental checks: a systematic review and economic evaluation.

   By Davenport C, Elley K, Salas C, Taylor-Weetman CL, Fry-Smith A, Bryan S, et al.

8. A multicentre randomised controlled trial assessing the costs and benefits of using structured information and analysis of women’s preferences in the management of menorrhagia.

   By Kennedy ADM, Sculpher MJ, Coulter A, Dwyer N, Rees M, Horsley S, et al.

9. Clinical effectiveness and cost–utility of photodynamic therapy for wet age-related macular degeneration: a systematic review and economic evaluation.

   By Meads C, Salas C, Roberts T, Moore D, Fry-Smith A, Hyde C.

10. Evaluation of molecular tests for prenatal diagnosis of chromosome abnormalities.

    By Grimshaw GM, Szczepura A, Hultén M, MacDonald F, Nevin NC, Sutton F, et al.

11. First and second trimester antenatal screening for Down’s syndrome: the results of the Serum, Urine and Ultrasound Screening Study (SURUSS).

    By Wald NJ, Rodeck C, Hackshaw AK, Walters J, Chitty L, Mackinson AM.

12. The effectiveness and cost-effectiveness of ultrasound locating devices for central venous access: a systematic review and economic evaluation.

    By Calvert N, Hind D, McWilliams RG, Thomas SM, Beverley C, Davidson A.

13. A systematic review of atypical antipsychotics in schizophrenia.

    By Bagnall A-M, Jones L, Lewis R, Ginnelly L, Glanville J, Torgerson D, et al.

14. Prostate Testing for Cancer and Treatment (ProtecT) feasibility study.

    By Donovan J, Hamdy F, Neal D, Peters T, Oliver S, Brindle L, et al.

15. Early thrombolysis for the treatment of acute myocardial infarction: a systematic review and economic evaluation.

    By Boland A, Dundar Y, Bagust A, Haycox A, Hill R, Mujica Mota R, et al.

16. Screening for fragile X syndrome: a literature review and modelling.

    By Song FJ, Barton P, Sleightholme V, Yao GL, Fry-Smith A.

17. Systematic review of endoscopic sinus surgery for nasal polyps.

    By Dalziel K, Stein K, Round A, Garside R, Royle P.

18. Towards efficient guidelines: how to monitor guideline use in primary care.

    By Hutchinson A, McIntosh A, Cox S, Gilbert C.

19. Effectiveness and cost-effectiveness of acute hospital-based spinal cord injuries services: systematic review.

    By Bagnall A-M, Jones L, Richardson G, Duffy S, Riemsma R.

20. Prioritisation of health technology assessment. The PATHS model: methods and case studies.

    By Townsend J, Buxton M, Harper G.

21. Systematic review of the clinical effectiveness and cost-effectiveness of tension-free vaginal tape for treatment of urinary stress incontinence.

    By Cody J, Wyness L, Wallace S, Glazener C, Kilonzo M, Stearns S, et al.

22. The clinical and cost-effectiveness of patient education models for diabetes: a systematic review and economic evaluation.

    By Loveman E, Cave C, Green C, Royle P, Dunn N, Waugh N.

23. The role of modelling in prioritising and planning clinical trials.

    By Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P.

24. Cost–benefit evaluation of routine influenza immunisation in people 65–74 years of age.

    By Allsup S, Gosney M, Haycox A, Regan M.

25. The clinical and cost-effectiveness of pulsatile machine perfusion versus cold storage of kidneys for transplantation retrieved from heart-beating and non-heart-beating donors.

    By Wight J, Chilcott J, Holmes M, Brewer N.

26. Can randomised trials rely on existing electronic data? A feasibility study to explore the value of routine data in health technology assessment.

    By Williams JG, Cheung WY, Cohen DR, Hutchings HA, Longo MF, Russell IT.

27. Evaluating non-randomised intervention studies.

    By Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al.

28. A randomised controlled trial to assess the impact of a package comprising a patient-orientated, evidence-based self- help guidebook and patient-centred consultations on disease management and satisfaction in inflammatory bowel disease.

    By Kennedy A, Nelson E, Reeves D, Richardson G, Roberts C, Robinson A, et al.

29. The effectiveness of diagnostic tests for the assessment of shoulder pain due to soft tissue disorders: a systematic review.

    By Dinnes J, Loveman E, McIntyre L, Waugh N.

30. The value of digital imaging in diabetic retinopathy.

    By Sharp PF, Olson J, Strachan F, Hipwell J, Ludbrook A, O’Donnell M, et al.

31. Lowering blood pressure to prevent myocardial infarction and stroke: a new preventive strategy.

    By Law M, Wald N, Morris J.

32. Clinical and cost-effectiveness of capecitabine and tegafur with uracil for the treatment of metastatic colorectal cancer: systematic review and economic evaluation.

    By Ward S, Kaltenthaler E, Cowan J, Brewer N.

33. Clinical and cost-effectiveness of new and emerging technologies for early localised prostate cancer: a systematic review.

    By Hummel S, Paisley S, Morgan A, Currie E, Brewer N.

34. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system.

    By Royle P, Waugh N.

35. Systematic review and economic decision modelling for the prevention and treatment of influenza A and B.

    By Turner D, Wailoo A, Nicholson K, Cooper N, Sutton A, Abrams K.

36. A randomised controlled trial to evaluate the clinical and cost-effectiveness of Hickman line insertions in adult cancer patients by nurses.

    By Boland A, Haycox A, Bagust A, Fitzsimmons L.

37. Redesigning postnatal care: a randomised controlled trial of protocol-based midwifery-led care focused on individual women’s physical and psychological health needs.

    By MacArthur C, Winter HR, Bick DE, Lilford RJ, Lancashire RJ, Knowles H, et al.

38. Estimating implied rates of discount in healthcare decision-making.

    By West RR, McNabb R, Thompson AGH, Sheldon TA, Grimley Evans J.

39. Systematic review of isolation policies in the hospital management of methicillin-resistant Staphylococcus aureus: a review of the literature with epidemiological and economic modelling.

    By Cooper BS, Stone SP, Kibbler CC, Cookson BD, Roberts JA, Medley GF, et al.

40. Treatments for spasticity and pain in multiple sclerosis: a systematic review.

    By Beard S, Hunn A, Wight J.

41. The inclusion of reports of randomised trials published in languages other than English in systematic reviews.

    By Moher D, Pham B, Lawson ML, Klassen TP.

42. The impact of screening on future health-promoting behaviours and health beliefs: a systematic review.

    By Bankhead CR, Brett J, Bukach C, Webster P, Stewart-Brown S, Munafo M, et al.

1. What is the best imaging strategy for acute stroke?

   By Wardlaw JM, Keir SL, Seymour J, Lewis S, Sandercock PAG, Dennis MS, et al.

2. Systematic review and modelling of the investigation of acute and chronic chest pain presenting in primary care.

   By Mant J, McManus RJ, Oakes RAL, Delaney BC, Barton PM, Deeks JJ, et al.

3. The effectiveness and cost-effectiveness of microwave and thermal balloon endometrial ablation for heavy menstrual bleeding: a systematic review and economic modelling.

   By Garside R, Stein K, Wyatt K, Round A, Price A.

4. A systematic review of the role of bisphosphonates in metastatic disease.

   By Ross JR, Saunders Y, Edmonds PM, Patel S, Wonderling D, Normand C, et al.

5. Systematic review of the clinical effectiveness and cost-effectiveness of capecitabine (Xeloda^®) for locally advanced and/or metastatic breast cancer.

   By Jones L, Hawkins N, Westwood M, Wright K, Richardson G, Riemsma R.

6. Effectiveness and efficiency of guideline dissemination and implementation strategies.

   By Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al.

7. Clinical effectiveness and costs of the Sugarbaker procedure for the treatment of pseudomyxoma peritonei.

   By Bryant J, Clegg AJ, Sidhu MK, Brodin H, Royle P, Davidson P.

8. Psychological treatment for insomnia in the regulation of long-term hypnotic drug use.

   By Morgan K, Dixon S, Mathers N, Thompson J, Tomeny M.

9. Improving the evaluation of therapeutic interventions in multiple sclerosis: development of a patient-based measure of outcome.

   By Hobart JC, Riazi A, Lamping DL, Fitzpatrick R, Thompson AJ.

10. A systematic review and economic evaluation of magnetic resonance cholangiopancreatography compared with diagnostic endoscopic retrograde cholangiopancreatography.

    By Kaltenthaler E, Bravo Vergel Y, Chilcott J, Thomas S, Blakeborough T, Walters SJ, et al.

11. The use of modelling to evaluate new drugs for patients with a chronic condition: the case of antibodies against tumour necrosis factor in rheumatoid arthritis.

    By Barton P, Jobanputra P, Wilson J, Bryan S, Burls A.

12. Clinical effectiveness and cost-effectiveness of neonatal screening for inborn errors of metabolism using tandem mass spectrometry: a systematic review.

    By Pandor A, Eastham J, Beverley C, Chilcott J, Paisley S.

13. Clinical effectiveness and cost-effectiveness of pioglitazone and rosiglitazone in the treatment of type 2 diabetes: a systematic review and economic evaluation.

    By Czoski-Murray C, Warren E, Chilcott J, Beverley C, Psyllaki MA, Cowan J.

14. Routine examination of the newborn: the EMREN study. Evaluation of an extension of the midwife role including a randomised controlled trial of appropriately trained midwives and paediatric senior house officers.

    By Townsend J, Wolke D, Hayes J, Davé S, Rogers C, Bloomfield L, et al.

15. Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach.

    By Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, et al.

16. A multi-centre randomised controlled trial of minimally invasive direct coronary bypass grafting versus percutaneous transluminal coronary angioplasty with stenting for proximal stenosis of the left anterior descending coronary artery.

    By Reeves BC, Angelini GD, Bryan AJ, Taylor FC, Cripps T, Spyt TJ, et al.

17. Does early magnetic resonance imaging influence management or improve outcome in patients referred to secondary care with low back pain? A pragmatic randomised controlled trial.

    By Gilbert FJ, Grant AM, Gillan MGC, Vale L, Scott NW, Campbell MK, et al.

18. The clinical and cost-effectiveness of anakinra for the treatment of rheumatoid arthritis in adults: a systematic review and economic analysis.

    By Clark W, Jobanputra P, Barton P, Burls A.

19. A rapid and systematic review and economic evaluation of the clinical and cost-effectiveness of newer drugs for treatment of mania associated with bipolar affective disorder.

    By Bridle C, Palmer S, Bagnall A-M, Darba J, Duffy S, Sculpher M, et al.

20. Liquid-based cytology in cervical screening: an updated rapid and systematic review and economic analysis.

    By Karnon J, Peters J, Platt J, Chilcott J, McGoogan E, Brewer N.

21. Systematic review of the long-term effects and economic consequences of treatments for obesity and implications for health improvement.

    By Avenell A, Broom J, Brown TJ, Poobalan A, Aucott L, Stearns SC, et al.

22. Autoantibody testing in children with newly diagnosed type 1 diabetes mellitus.

    By Dretzke J, Cummins C, Sandercock J, Fry-Smith A, Barrett T, Burls A.

23. Clinical effectiveness and cost-effectiveness of prehospital intravenous fluids in trauma patients.

    By Dretzke J, Sandercock J, Bayliss S, Burls A.

24. Newer hypnotic drugs for the short-term management of insomnia: a systematic review and economic evaluation.

    By Dündar Y, Boland A, Strobl J, Dodd S, Haycox A, Bagust A, et al.

25. Development and validation of methods for assessing the quality of diagnostic accuracy studies.

    By Whiting P, Rutjes AWS, Dinnes J, Reitsma JB, Bossuyt PMM, Kleijnen J.

26. EVALUATE hysterectomy trial: a multicentre randomised trial comparing abdominal, vaginal and laparoscopic methods of hysterectomy.

    By Garry R, Fountain J, Brown J, Manca A, Mason S, Sculpher M, et al.

27. Methods for expected value of information analysis in complex health economic models: developments on the health economics of interferon-β and glatiramer acetate for multiple sclerosis.

    By Tappenden P, Chilcott JB, Eggington S, Oakley J, McCabe C.

28. Effectiveness and cost-effectiveness of imatinib for first-line treatment of chronic myeloid leukaemia in chronic phase: a systematic review and economic analysis.

    By Dalziel K, Round A, Stein K, Garside R, Price A.

29. VenUS I: a randomised controlled trial of two types of bandage for treating venous leg ulcers.

    By Iglesias C, Nelson EA, Cullum NA, Torgerson DJ, on behalf of the VenUS Team.

30. Systematic review of the effectiveness and cost-effectiveness, and economic evaluation, of myocardial perfusion scintigraphy for the diagnosis and management of angina and myocardial infarction.

    By Mowatt G, Vale L, Brazzelli M, Hernandez R, Murray A, Scott N, et al.

31. A pilot study on the use of decision theory and value of information analysis as part of the NHS Health Technology Assessment programme.

    By Claxton K, Ginnelly L, Sculpher M, Philips Z, Palmer S.

32. The Social Support and Family Health Study: a randomised controlled trial and economic evaluation of two alternative forms of postnatal support for mothers living in disadvantaged inner-city areas.

    By Wiggins M, Oakley A, Roberts I, Turner H, Rajan L, Austerberry H, et al.

33. Psychosocial aspects of genetic screening of pregnant women and newborns: a systematic review.

    By Green JM, Hewison J, Bekker HL, Bryant, Cuckle HS.

34. Evaluation of abnormal uterine bleeding: comparison of three outpatient procedures within cohorts defined by age and menopausal status.

    By Critchley HOD, Warner P, Lee AJ, Brechin S, Guise J, Graham B.

35. Coronary artery stents: a rapid systematic review and economic evaluation.

    By Hill R, Bagust A, Bakhai A, Dickson R, Dündar Y, Haycox A, et al.

36. Review of guidelines for good practice in decision-analytic modelling in health technology assessment.

    By Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al.

37. Rituximab (MabThera^®) for aggressive non-Hodgkin’s lymphoma: systematic review and economic evaluation.

    By Knight C, Hind D, Brewer N, Abbott V.

38. Clinical effectiveness and cost-effectiveness of clopidogrel and modified-release dipyridamole in the secondary prevention of occlusive vascular events: a systematic review and economic evaluation.

    By Jones L, Griffin S, Palmer S, Main C, Orton V, Sculpher M, et al.

39. Pegylated interferon α-2a and -2b in combination with ribavirin in the treatment of chronic hepatitis C: a systematic review and economic evaluation.

    By Shepherd J, Brodin H, Cave C, Waugh N, Price A, Gabbay J.

40. Clopidogrel used in combination with aspirin compared with aspirin alone in the treatment of non-ST-segment- elevation acute coronary syndromes: a systematic review and economic evaluation.

    By Main C, Palmer S, Griffin S, Jones L, Orton V, Sculpher M, et al.

41. Provision, uptake and cost of cardiac rehabilitation programmes: improving services to under-represented groups.

    By Beswick AD, Rees K, Griebsch I, Taylor FC, Burke M, West RR, et al.

42. Involving South Asian patients in clinical trials.

    By Hussain-Gambles M, Leese B, Atkin K, Brown J, Mason S, Tovey P.

43. Clinical and cost-effectiveness of continuous subcutaneous insulin infusion for diabetes.

    By Colquitt JL, Green C, Sidhu MK, Hartwell D, Waugh N.

44. Identification and assessment of ongoing trials in health technology assessment reviews.

    By Song FJ, Fry-Smith A, Davenport C, Bayliss S, Adi Y, Wilson JS, et al.

45. Systematic review and economic evaluation of a long-acting insulin analogue, insulin glargine

    By Warren E, Weatherley-Jones E, Chilcott J, Beverley C.

46. Supplementation of a home-based exercise programme with a class-based programme for people with osteoarthritis of the knees: a randomised controlled trial and health economic analysis.

    By McCarthy CJ, Mills PM, Pullen R, Richardson G, Hawkins N, Roberts CR, et al.

47. Clinical and cost-effectiveness of once-daily versus more frequent use of same potency topical corticosteroids for atopic eczema: a systematic review and economic evaluation.

    By Green C, Colquitt JL, Kirby J, Davidson P, Payne E.

48. Acupuncture of chronic headache disorders in primary care: randomised controlled trial and economic analysis.

    By Vickers AJ, Rees RW, Zollman CE, McCarney R, Smith CM, Ellis N, et al.

49. Generalisability in economic evaluation studies in healthcare: a review and case studies.

    By Sculpher MJ, Pang FS, Manca A, Drummond MF, Golder S, Urdahl H, et al.

50. Virtual outreach: a randomised controlled trial and economic evaluation of joint teleconferenced medical consultations.

    By Wallace P, Barber J, Clayton W, Currell R, Fleming K, Garner P, et al.

1. Randomised controlled multiple treatment comparison to provide a cost-effectiveness rationale for the selection of antimicrobial therapy in acne.

   By Ozolins M, Eady EA, Avery A, Cunliffe WJ, O’Neill C, Simpson NB, et al.

2. Do the findings of case series studies vary significantly according to methodological characteristics?

   By Dalziel K, Round A, Stein K, Garside R, Castelnuovo E, Payne L.

3. Improving the referral process for familial breast cancer genetic counselling: findings of three randomised controlled trials of two interventions.

   By Wilson BJ, Torrance N, Mollison J, Wordsworth S, Gray JR, Haites NE, et al.

4. Randomised evaluation of alternative electrosurgical modalities to treat bladder outflow obstruction in men with benign prostatic hyperplasia.

   By Fowler C, McAllister W, Plail R, Karim O, Yang Q.

5. A pragmatic randomised controlled trial of the cost-effectiveness of palliative therapies for patients with inoperable oesophageal cancer.

   By Shenfine J, McNamee P, Steen N, Bond J, Griffin SM.

6. Impact of computer-aided detection prompts on the sensitivity and specificity of screening mammography.

   By Taylor P, Champness J, Given- Wilson R, Johnston K, Potts H.

7. Issues in data monitoring and interim analysis of trials.

   By Grant AM, Altman DG, Babiker AB, Campbell MK, Clemens FJ, Darbyshire JH, et al.

8. Lay public’s understanding of equipoise and randomisation in randomised controlled trials.

   By Robinson EJ, Kerr CEP, Stevens AJ, Lilford RJ, Braunholtz DA, Edwards SJ, et al.

9. Clinical and cost-effectiveness of electroconvulsive therapy for depressive illness, schizophrenia, catatonia and mania: systematic reviews and economic modelling studies.

   By Greenhalgh J, Knight C, Hind D, Beverley C, Walters S.

10. Measurement of health-related quality of life for people with dementia: development of a new instrument (DEMQOL) and an evaluation of current methodology.

    By Smith SC, Lamping DL, Banerjee S, Harwood R, Foley B, Smith P, et al.

11. Clinical effectiveness and cost-effectiveness of drotrecogin alfa (activated) (Xigris^®) for the treatment of severe sepsis in adults: a systematic review and economic evaluation.

    By Green C, Dinnes J, Takeda A, Shepherd J, Hartwell D, Cave C, et al.

12. A methodological review of how heterogeneity has been examined in systematic reviews of diagnostic test accuracy.

    By Dinnes J, Deeks J, Kirby J, Roderick P.

13. Cervical screening programmes: can automation help? Evidence from systematic reviews, an economic analysis and a simulation modelling exercise applied to the UK.

    By Willis BH, Barton P, Pearmain P, Bryan S, Hyde C.

14. Laparoscopic surgery for inguinal hernia repair: systematic review of effectiveness and economic evaluation.

    By McCormack K, Wake B, Perez J, Fraser C, Cook J, McIntosh E, et al.

15. Clinical effectiveness, tolerability and cost-effectiveness of newer drugs for epilepsy in adults: a systematic review and economic evaluation.

    By Wilby J, Kainth A, Hawkins N, Epstein D, McIntosh H, McDaid C, et al.

16. A randomised controlled trial to compare the cost-effectiveness of tricyclic antidepressants, selective serotonin reuptake inhibitors and lofepramine.

    By Peveler R, Kendrick T, Buxton M, Longworth L, Baldwin D, Moore M, et al.

17. Clinical effectiveness and cost-effectiveness of immediate angioplasty for acute myocardial infarction: systematic review and economic evaluation.

    By Hartwell D, Colquitt J, Loveman E, Clegg AJ, Brodin H, Waugh N, et al.

18. A randomised controlled comparison of alternative strategies in stroke care.

    By Kalra L, Evans A, Perez I, Knapp M, Swift C, Donaldson N.

19. The investigation and analysis of critical incidents and adverse events in healthcare.

    By Woloshynowych M, Rogers S, Taylor-Adams S, Vincent C.

20. Potential use of routine databases in health technology assessment.

    By Raftery J, Roderick P, Stevens A.

21. Clinical and cost-effectiveness of newer immunosuppressive regimens in renal transplantation: a systematic review and modelling study.

    By Woodroffe R, Yao GL, Meads C, Bayliss S, Ready A, Raftery J, et al.

22. A systematic review and economic evaluation of alendronate, etidronate, risedronate, raloxifene and teriparatide for the prevention and treatment of postmenopausal osteoporosis.

    By Stevenson M, Lloyd Jones M, De Nigris E, Brewer N, Davis S, Oakley J.

23. A systematic review to examine the impact of psycho-educational interventions on health outcomes and costs in adults and children with difficult asthma.

    By Smith JR, Mugford M, Holland R, Candy B, Noble MJ, Harrison BDW, et al.

24. An evaluation of the costs, effectiveness and quality of renal replacement therapy provision in renal satellite units in England and Wales.

    By Roderick P, Nicholson T, Armitage A, Mehta R, Mullee M, Gerard K, et al.

25. Imatinib for the treatment of patients with unresectable and/or metastatic gastrointestinal stromal tumours: systematic review and economic evaluation.

    By Wilson J, Connock M, Song F, Yao G, Fry-Smith A, Raftery J, et al.

26. Indirect comparisons of competing interventions.

    By Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R, et al.

27. Cost-effectiveness of alternative strategies for the initial medical management of non-ST elevation acute coronary syndrome: systematic review and decision-analytical modelling.

    By Robinson M, Palmer S, Sculpher M, Philips Z, Ginnelly L, Bowens A, et al.

28. Outcomes of electrically stimulated gracilis neosphincter surgery.

    By Tillin T, Chambers M, Feldman R.

29. The effectiveness and cost-effectiveness of pimecrolimus and tacrolimus for atopic eczema: a systematic review and economic evaluation.

    By Garside R, Stein K, Castelnuovo E, Pitt M, Ashcroft D, Dimmock P, et al.

30. Systematic review on urine albumin testing for early detection of diabetic complications.

    By Newman DJ, Mattock MB, Dawnay ABS, Kerry S, McGuire A, Yaqoob M, et al.

31. Randomised controlled trial of the cost-effectiveness of water-based therapy for lower limb osteoarthritis.

    By Cochrane T, Davey RC, Matthes Edwards SM.

32. Longer term clinical and economic benefits of offering acupuncture care to patients with chronic low back pain.

    By Thomas KJ, MacPherson H, Ratcliffe J, Thorpe L, Brazier J, Campbell M, et al.

33. Cost-effectiveness and safety of epidural steroids in the management of sciatica.

    By Price C, Arden N, Coglan L, Rogers P.

34. The British Rheumatoid Outcome Study Group (BROSG) randomised controlled trial to compare the effectiveness and cost-effectiveness of aggressive versus symptomatic therapy in established rheumatoid arthritis.

    By Symmons D, Tricker K, Roberts C, Davies L, Dawes P, Scott DL.

35. Conceptual framework and systematic review of the effects of participants’ and professionals’ preferences in randomised controlled trials.

    By King M, Nazareth I, Lampe F, Bower P, Chandler M, Morou M, et al.

36. The clinical and cost-effectiveness of implantable cardioverter defibrillators: a systematic review.

    By Bryant J, Brodin H, Loveman E, Payne E, Clegg A.

37. A trial of problem-solving by community mental health nurses for anxiety, depression and life difficulties among general practice patients. The CPN-GP study.

    By Kendrick T, Simons L, Mynors-Wallis L, Gray A, Lathlean J, Pickering R, et al.

38. The causes and effects of socio-demographic exclusions from clinical trials.

    By Bartlett C, Doyal L, Ebrahim S, Davey P, Bachmann M, Egger M, et al.

39. Is hydrotherapy cost-effective? A randomised controlled trial of combined hydrotherapy programmes compared with physiotherapy land techniques in children with juvenile idiopathic arthritis.

    By Epps H, Ginnelly L, Utley M, Southwood T, Gallivan S, Sculpher M, et al.

40. A randomised controlled trial and cost-effectiveness study of systematic screening (targeted and total population screening) versus routine practice for the detection of atrial fibrillation in people aged 65 and over. The SAFE study.

    By Hobbs FDR, Fitzmaurice DA, Mant J, Murray E, Jowett S, Bryan S, et al.

41. Displaced intracapsular hip fractures in fit, older people: a randomised comparison of reduction and fixation, bipolar hemiarthroplasty and total hip arthroplasty.

    By Keating JF, Grant A, Masson M, Scott NW, Forbes JF.

42. Long-term outcome of cognitive behaviour therapy clinical trials in central Scotland.

    By Durham RC, Chambers JA, Power KG, Sharp DM, Macdonald RR, Major KA, et al.

43. The effectiveness and cost-effectiveness of dual-chamber pacemakers compared with single-chamber pacemakers for bradycardia due to atrioventricular block or sick sinus syndrome: systematic review and economic evaluation.

    By Castelnuovo E, Stein K, Pitt M, Garside R, Payne E.

44. Newborn screening for congenital heart defects: a systematic review and cost-effectiveness analysis.

    By Knowles R, Griebsch I, Dezateux C, Brown J, Bull C, Wren C.

45. The clinical and cost-effectiveness of left ventricular assist devices for end-stage heart failure: a systematic review and economic evaluation.

    By Clegg AJ, Scott DA, Loveman E, Colquitt J, Hutchinson J, Royle P, et al.

46. The effectiveness of the Heidelberg Retina Tomograph and laser diagnostic glaucoma scanning system (GDx) in detecting and monitoring glaucoma.

    By Kwartz AJ, Henson DB, Harper RA, Spencer AF, McLeod D.

47. Clinical and cost-effectiveness of autologous chondrocyte implantation for cartilage defects in knee joints: systematic review and economic evaluation.

    By Clar C, Cummins E, McIntyre L, Thomas S, Lamb J, Bain L, et al.

48. Systematic review of effectiveness of different treatments for childhood retinoblastoma.

    By McDaid C, Hartley S, Bagnall A-M, Ritchie G, Light K, Riemsma R.

49. Towards evidence-based guidelines for the prevention of venous thromboembolism: systematic reviews of mechanical methods, oral anticoagulation, dextran and regional anaesthesia as thromboprophylaxis.

    By Roderick P, Ferris G, Wilson K, Halls H, Jackson D, Collins R, et al.

50. The effectiveness and cost-effectiveness of parent training/education programmes for the treatment of conduct disorder, including oppositional defiant disorder, in children.

    By Dretzke J, Frew E, Davenport C, Barlow J, Stewart-Brown S, Sandercock J, et al.

1. The clinical and cost-effectiveness of donepezil, rivastigmine, galantamine and memantine for Alzheimer’s disease.

   By Loveman E, Green C, Kirby J, Takeda A, Picot J, Payne E, et al.

2. FOOD: a multicentre randomised trial evaluating feeding policies in patients admitted to hospital with a recent stroke.

   By Dennis M, Lewis S, Cranswick G, Forbes J.

3. The clinical effectiveness and cost-effectiveness of computed tomography screening for lung cancer: systematic reviews.

   By Black C, Bagust A, Boland A, Walker S, McLeod C, De Verteuil R, et al.

4. A systematic review of the effectiveness and cost-effectiveness of neuroimaging assessments used to visualise the seizure focus in people with refractory epilepsy being considered for surgery.

   By Whiting P, Gupta R, Burch J, Mujica Mota RE, Wright K, Marson A, et al.

5. Comparison of conference abstracts and presentations with full-text articles in the health technology assessments of rapidly evolving technologies.

   By Dundar Y, Dodd S, Dickson R, Walley T, Haycox A, Williamson PR.

6. Systematic review and evaluation of methods of assessing urinary incontinence.

   By Martin JL, Williams KS, Abrams KR, Turner DA, Sutton AJ, Chapple C, et al.

7. The clinical effectiveness and cost-effectiveness of newer drugs for children with epilepsy. A systematic review.

   By Connock M, Frew E, Evans B-W, Bryan S, Cummins C, Fry-Smith A, et al.

8. Surveillance of Barrett’s oesophagus: exploring the uncertainty through systematic review, expert workshop and economic modelling.

   By Garside R, Pitt M, Somerville M, Stein K, Price A, Gilbert N.

9. Topotecan, pegylated liposomal doxorubicin hydrochloride and paclitaxel for second-line or subsequent treatment of advanced ovarian cancer: a systematic review and economic evaluation.

   By Main C, Bojke L, Griffin S, Norman G, Barbieri M, Mather L, et al.

10. Evaluation of molecular techniques in prediction and diagnosis of cytomegalovirus disease in immunocompromised patients.

    By Szczepura A, Westmoreland D, Vinogradova Y, Fox J, Clark M.

11. Screening for thrombophilia in high-risk situations: systematic review and cost-effectiveness analysis. The Thrombosis: Risk and Economic Assessment of Thrombophilia Screening (TREATS) study.

    By Wu O, Robertson L, Twaddle S, Lowe GDO, Clark P, Greaves M, et al.

12. A series of systematic reviews to inform a decision analysis for sampling and treating infected diabetic foot ulcers.

    By Nelson EA, O’Meara S, Craig D, Iglesias C, Golder S, Dalton J, et al.

13. Randomised clinical trial, observational study and assessment of cost-effectiveness of the treatment of varicose veins (REACTIV trial).

    By Michaels JA, Campbell WB, Brazier JE, MacIntyre JB, Palfreyman SJ, Ratcliffe J, et al.

14. The cost-effectiveness of screening for oral cancer in primary care.

    By Speight PM, Palmer S, Moles DR, Downer MC, Smith DH, Henriksson M, et al.

15. Measurement of the clinical and cost-effectiveness of non-invasive diagnostic testing strategies for deep vein thrombosis.

    By Goodacre S, Sampson F, Stevenson M, Wailoo A, Sutton A, Thomas S, et al.

16. Systematic review of the effectiveness and cost-effectiveness of HealOzone^® for the treatment of occlusal pit/fissure caries and root caries.

    By Brazzelli M, McKenzie L, Fielding S, Fraser C, Clarkson J, Kilonzo M, et al.

17. Randomised controlled trials of conventional antipsychotic versus new atypical drugs, and new atypical drugs versus clozapine, in people with schizophrenia responding poorly to, or intolerant of, current drug treatment.

    By Lewis SW, Davies L, Jones PB, Barnes TRE, Murray RM, Kerwin R, et al.

18. Diagnostic tests and algorithms used in the investigation of haematuria: systematic reviews and economic evaluation.

    By Rodgers M, Nixon J, Hempel S, Aho T, Kelly J, Neal D, et al.

19. Cognitive behavioural therapy in addition to antispasmodic therapy for irritable bowel syndrome in primary care: randomised controlled trial.

    By Kennedy TM, Chalder T, McCrone P, Darnley S, Knapp M, Jones RH, et al.

20. A systematic review of the clinical effectiveness and cost-effectiveness of enzyme replacement therapies for Fabry’s disease and mucopolysaccharidosis type 1.

    By Connock M, Juarez-Garcia A, Frew E, Mans A, Dretzke J, Fry-Smith A, et al.

21. Health benefits of antiviral therapy for mild chronic hepatitis C: randomised controlled trial and economic evaluation.

    By Wright M, Grieve R, Roberts J, Main J, Thomas HC, on behalf of the UK Mild Hepatitis C Trial Investigators.

22. Pressure relieving support surfaces: a randomised evaluation.

    By Nixon J, Nelson EA, Cranny G, Iglesias CP, Hawkins K, Cullum NA, et al.

23. A systematic review and economic model of the effectiveness and cost-effectiveness of methylphenidate, dexamfetamine and atomoxetine for the treatment of attention deficit hyperactivity disorder in children and adolescents.

    By King S, Griffin S, Hodges Z, Weatherly H, Asseburg C, Richardson G, et al.

24. The clinical effectiveness and cost-effectiveness of enzyme replacement therapy for Gaucher’s disease: a systematic review.

    By Connock M, Burls A, Frew E, Fry-Smith A, Juarez-Garcia A, McCabe C, et al.

25. Effectiveness and cost-effectiveness of salicylic acid and cryotherapy for cutaneous warts. An economic decision model.

    By Thomas KS, Keogh-Brown MR, Chalmers JR, Fordham RJ, Holland RC, Armstrong SJ, et al.

26. A systematic literature review of the effectiveness of non-pharmacological interventions to prevent wandering in dementia and evaluation of the ethical implications and acceptability of their use.

    By Robinson L, Hutchings D, Corner L, Beyer F, Dickinson H, Vanoli A, et al.

27. A review of the evidence on the effects and costs of implantable cardioverter defibrillator therapy in different patient groups, and modelling of cost-effectiveness and cost–utility for these groups in a UK context.

    By Buxton M, Caine N, Chase D, Connelly D, Grace A, Jackson C, et al.

28. Adefovir dipivoxil and pegylated interferon alfa-2a for the treatment of chronic hepatitis B: a systematic review and economic evaluation.

    By Shepherd J, Jones J, Takeda A, Davidson P, Price A.

29. An evaluation of the clinical and cost-effectiveness of pulmonary artery catheters in patient management in intensive care: a systematic review and a randomised controlled trial.

    By Harvey S, Stevens K, Harrison D, Young D, Brampton W, McCabe C, et al.

30. Accurate, practical and cost-effective assessment of carotid stenosis in the UK.

    By Wardlaw JM, Chappell FM, Stevenson M, De Nigris E, Thomas S, Gillard J, et al.

31. Etanercept and infliximab for the treatment of psoriatic arthritis: a systematic review and economic evaluation.

    By Woolacott N, Bravo Vergel Y, Hawkins N, Kainth A, Khadjesari Z, Misso K, et al.

32. The cost-effectiveness of testing for hepatitis C in former injecting drug users.

    By Castelnuovo E, Thompson-Coon J, Pitt M, Cramp M, Siebert U, Price A, et al.

33. Computerised cognitive behaviour therapy for depression and anxiety update: a systematic review and economic evaluation.

    By Kaltenthaler E, Brazier J, De Nigris E, Tumur I, Ferriter M, Beverley C, et al.

34. Cost-effectiveness of using prognostic information to select women with breast cancer for adjuvant systemic therapy.

    By Williams C, Brunskill S, Altman D, Briggs A, Campbell H, Clarke M, et al.

35. Psychological therapies including dialectical behaviour therapy for borderline personality disorder: a systematic review and preliminary economic evaluation.

    By Brazier J, Tumur I, Holmes M, Ferriter M, Parry G, Dent-Brown K, et al.

36. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model.

    By Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al.

37. Cognitive behavioural therapy in chronic fatigue syndrome: a randomised controlled trial of an outpatient group programme.

    By O’Dowd H, Gladwell P, Rogers CA, Hollinghurst S, Gregory A.

38. A comparison of the cost-effectiveness of five strategies for the prevention of nonsteroidal anti-inflammatory drug-induced gastrointestinal toxicity: a systematic review with economic modelling.

    By Brown TJ, Hooper L, Elliott RA, Payne K, Webb R, Roberts C, et al.

39. The effectiveness and cost-effectiveness of computed tomography screening for coronary artery disease: systematic review.

    By Waugh N, Black C, Walker S, McIntyre L, Cummins E, Hillis G.

40. What are the clinical outcome and cost-effectiveness of endoscopy undertaken by nurses when compared with doctors? A Multi-Institution Nurse Endoscopy Trial (MINuET).

    By Williams J, Russell I, Durai D, Cheung W-Y, Farrin A, Bloor K, et al.

41. The clinical and cost-effectiveness of oxaliplatin and capecitabine for the adjuvant treatment of colon cancer: systematic review and economic evaluation.

    By Pandor A, Eggington S, Paisley S, Tappenden P, Sutcliffe P.

42. A systematic review of the effectiveness of adalimumab, etanercept and infliximab for the treatment of rheumatoid arthritis in adults and an economic evaluation of their cost-effectiveness.

    By Chen Y-F, Jobanputra P, Barton P, Jowett S, Bryan S, Clark W, et al.

43. Telemedicine in dermatology: a randomised controlled trial.

    By Bowns IR, Collins K, Walters SJ, McDonagh AJG.

44. Cost-effectiveness of cell salvage and alternative methods of minimising perioperative allogeneic blood transfusion: a systematic review and economic model.

    By Davies L, Brown TJ, Haynes S, Payne K, Elliott RA, McCollum C.

45. Clinical effectiveness and cost-effectiveness of laparoscopic surgery for colorectal cancer: systematic reviews and economic evaluation.

    By Murray A, Lourenco T, de Verteuil R, Hernandez R, Fraser C, McKinley A, et al.

46. Etanercept and efalizumab for the treatment of psoriasis: a systematic review.

    By Woolacott N, Hawkins N, Mason A, Kainth A, Khadjesari Z, Bravo Vergel Y, et al.

47. Systematic reviews of clinical decision tools for acute abdominal pain.

    By Liu JLY, Wyatt JC, Deeks JJ, Clamp S, Keen J, Verde P, et al.

48. Evaluation of the ventricular assist device programme in the UK.

    By Sharples L, Buxton M, Caine N, Cafferty F, Demiris N, Dyer M, et al.

49. A systematic review and economic model of the clinical and cost-effectiveness of immunosuppressive therapy for renal transplantation in children.

    By Yao G, Albon E, Adi Y, Milford D, Bayliss S, Ready A, et al.

50. Amniocentesis results: investigation of anxiety. The ARIA trial.

    By Hewison J, Nixon J, Fountain J, Cocks K, Jones C, Mason G, et al.

1. Pemetrexed disodium for the treatment of malignant pleural mesothelioma: a systematic review and economic evaluation.

   By Dundar Y, Bagust A, Dickson R, Dodd S, Green J, Haycox A, et al.

2. A systematic review and economic model of the clinical effectiveness and cost-effectiveness of docetaxel in combination with prednisone or prednisolone for the treatment of hormone-refractory metastatic prostate cancer.

   By Collins R, Fenwick E, Trowman R, Perard R, Norman G, Light K, et al.

3. A systematic review of rapid diagnostic tests for the detection of tuberculosis infection.

   By Dinnes J, Deeks J, Kunst H, Gibson A, Cummins E, Waugh N, et al.

4. The clinical effectiveness and cost-effectiveness of strontium ranelate for the prevention of osteoporotic fragility fractures in postmenopausal women.

   By Stevenson M, Davis S, Lloyd-Jones M, Beverley C.

5. A systematic review of quantitative and qualitative research on the role and effectiveness of written information available to patients about individual medicines.

   By Raynor DK, Blenkinsopp A, Knapp P, Grime J, Nicolson DJ, Pollock K, et al.

6. Oral naltrexone as a treatment for relapse prevention in formerly opioid-dependent drug users: a systematic review and economic evaluation.

   By Adi Y, Juarez-Garcia A, Wang D, Jowett S, Frew E, Day E, et al.

7. Glucocorticoid-induced osteoporosis: a systematic review and cost–utility analysis.

   By Kanis JA, Stevenson M, McCloskey EV, Davis S, Lloyd-Jones M.

8. Epidemiological, social, diagnostic and economic evaluation of population screening for genital chlamydial infection.

   By Low N, McCarthy A, Macleod J, Salisbury C, Campbell R, Roberts TE, et al.

9. Methadone and buprenorphine for the management of opioid dependence: a systematic review and economic evaluation.

   By Connock M, Juarez-Garcia A, Jowett S, Frew E, Liu Z, Taylor RJ, et al.

10. Exercise Evaluation Randomised Trial (EXERT): a randomised trial comparing GP referral for leisure centre-based exercise, community-based walking and advice only.

    By Isaacs AJ, Critchley JA, See Tai S, Buckingham K, Westley D, Harridge SDR, et al.

11. Interferon alfa (pegylated and non-pegylated) and ribavirin for the treatment of mild chronic hepatitis C: a systematic review and economic evaluation.

    By Shepherd J, Jones J, Hartwell D, Davidson P, Price A, Waugh N.

12. Systematic review and economic evaluation of bevacizumab and cetuximab for the treatment of metastatic colorectal cancer.

    By Tappenden P, Jones R, Paisley S, Carroll C.

13. A systematic review and economic evaluation of epoetin alfa, epoetin beta and darbepoetin alfa in anaemia associated with cancer, especially that attributable to cancer treatment.

    By Wilson J, Yao GL, Raftery J, Bohlius J, Brunskill S, Sandercock J, et al.

14. A systematic review and economic evaluation of statins for the prevention of coronary events.

    By Ward S, Lloyd Jones M, Pandor A, Holmes M, Ara R, Ryan A, et al.

15. A systematic review of the effectiveness and cost-effectiveness of different models of community-based respite care for frail older people and their carers.

    By Mason A, Weatherly H, Spilsbury K, Arksey H, Golder S, Adamson J, et al.

16. Additional therapy for young children with spastic cerebral palsy: a randomised controlled trial.

    By Weindling AM, Cunningham CC, Glenn SM, Edwards RT, Reeves DJ.

17. Screening for type 2 diabetes: literature review and economic modelling.

    By Waugh N, Scotland G, McNamee P, Gillett M, Brennan A, Goyder E, et al.

18. The effectiveness and cost-effectiveness of cinacalcet for secondary hyperparathyroidism in end-stage renal disease patients on dialysis: a systematic review and economic evaluation.

    By Garside R, Pitt M, Anderson R, Mealing S, Roome C, Snaith A, et al.

19. The clinical effectiveness and cost-effectiveness of gemcitabine for metastatic breast cancer: a systematic review and economic evaluation.

    By Takeda AL, Jones J, Loveman E, Tan SC, Clegg AJ.

20. A systematic review of duplex ultrasound, magnetic resonance angiography and computed tomography angiography for the diagnosis and assessment of symptomatic, lower limb peripheral arterial disease.

    By Collins R, Cranny G, Burch J, Aguiar-Ibáñez R, Craig D, Wright K, et al.

21. The clinical effectiveness and cost-effectiveness of treatments for children with idiopathic steroid-resistant nephrotic syndrome: a systematic review.

    By Colquitt JL, Kirby J, Green C, Cooper K, Trompeter RS.

22. A systematic review of the routine monitoring of growth in children of primary school age to identify growth-related conditions.

    By Fayter D, Nixon J, Hartley S, Rithalia A, Butler G, Rudolf M, et al.

23. Systematic review of the effectiveness of preventing and treating Staphylococcus aureus carriage in reducing peritoneal catheter-related infections.

    By McCormack K, Rabindranath K, Kilonzo M, Vale L, Fraser C, McIntyre L, et al.

24. The clinical effectiveness and cost of repetitive transcranial magnetic stimulation versus electroconvulsive therapy in severe depression: a multicentre pragmatic randomised controlled trial and economic analysis.

    By McLoughlin DM, Mogg A, Eranti S, Pluck G, Purvis R, Edwards D, et al.

25. A randomised controlled trial and economic evaluation of direct versus indirect and individual versus group modes of speech and language therapy for children with primary language impairment.

    By Boyle J, McCartney E, Forbes J, O’Hare A.

26. Hormonal therapies for early breast cancer: systematic review and economic evaluation.

    By Hind D, Ward S, De Nigris E, Simpson E, Carroll C, Wyld L.

27. Cardioprotection against the toxic effects of anthracyclines given to children with cancer: a systematic review.

    By Bryant J, Picot J, Levitt G, Sullivan I, Baxter L, Clegg A.

28. Adalimumab, etanercept and infliximab for the treatment of ankylosing spondylitis: a systematic review and economic evaluation.

    By McLeod C, Bagust A, Boland A, Dagenais P, Dickson R, Dundar Y, et al.

29. Prenatal screening and treatment strategies to prevent group B streptococcal and other bacterial infections in early infancy: cost-effectiveness and expected value of information analyses.

    By Colbourn T, Asseburg C, Bojke L, Philips Z, Claxton K, Ades AE, et al.

30. Clinical effectiveness and cost-effectiveness of bone morphogenetic proteins in the non-healing of fractures and spinal fusion: a systematic review.

    By Garrison KR, Donell S, Ryder J, Shemilt I, Mugford M, Harvey I, et al.

31. A randomised controlled trial of postoperative radiotherapy following breast-conserving surgery in a minimum-risk older population. The PRIME trial.

    By Prescott RJ, Kunkler IH, Williams LJ, King CC, Jack W, van der Pol M, et al.

32. Current practice, accuracy, effectiveness and cost-effectiveness of the school entry hearing screen.

    By Bamford J, Fortnum H, Bristow K, Smith J, Vamvakas G, Davies L, et al.

33. The clinical effectiveness and cost-effectiveness of inhaled insulin in diabetes mellitus: a systematic review and economic evaluation.

    By Black C, Cummins E, Royle P, Philip S, Waugh N.

34. Surveillance of cirrhosis for hepatocellular carcinoma: systematic review and economic analysis.

    By Thompson Coon J, Rogers G, Hewson P, Wright D, Anderson R, Cramp M, et al.

35. The Birmingham Rehabilitation Uptake Maximisation Study (BRUM). Homebased compared with hospital-based cardiac rehabilitation in a multi-ethnic population: cost-effectiveness and patient adherence.

    By Jolly K, Taylor R, Lip GYH, Greenfield S, Raftery J, Mant J, et al.

36. A systematic review of the clinical, public health and cost-effectiveness of rapid diagnostic tests for the detection and identification of bacterial intestinal pathogens in faeces and food.

    By Abubakar I, Irvine L, Aldus CF, Wyatt GM, Fordham R, Schelenz S, et al.

37. A randomised controlled trial examining the longer-term outcomes of standard versus new antiepileptic drugs. The SANAD trial.

    By Marson AG, Appleton R, Baker GA, Chadwick DW, Doughty J, Eaton B, et al.

38. Clinical effectiveness and cost-effectiveness of different models of managing long-term oral anti-coagulation therapy: a systematic review and economic modelling.

    By Connock M, Stevens C, Fry-Smith A, Jowett S, Fitzmaurice D, Moore D, et al.

39. A systematic review and economic model of the clinical effectiveness and cost-effectiveness of interventions for preventing relapse in people with bipolar disorder.

    By Soares-Weiser K, Bravo Vergel Y, Beynon S, Dunn G, Barbieri M, Duffy S, et al.

40. Taxanes for the adjuvant treatment of early breast cancer: systematic review and economic evaluation.

    By Ward S, Simpson E, Davis S, Hind D, Rees A, Wilkinson A.

41. The clinical effectiveness and cost-effectiveness of screening for open angle glaucoma: a systematic review and economic evaluation.

    By Burr JM, Mowatt G, Hernández R, Siddiqui MAR, Cook J, Lourenco T, et al.

42. Acceptability, benefit and costs of early screening for hearing disability: a study of potential screening tests and models.

    By Davis A, Smith P, Ferguson M, Stephens D, Gianopoulos I.

43. Contamination in trials of educational interventions.

    By Keogh-Brown MR, Bachmann MO, Shepstone L, Hewitt C, Howe A, Ramsay CR, et al.

44. Overview of the clinical effectiveness of positron emission tomography imaging in selected cancers.

    By Facey K, Bradbury I, Laking G, Payne E.

45. The effectiveness and cost-effectiveness of carmustine implants and temozolomide for the treatment of newly diagnosed high-grade glioma: a systematic review and economic evaluation.

    By Garside R, Pitt M, Anderson R, Rogers G, Dyer M, Mealing S, et al.

46. Drug-eluting stents: a systematic review and economic evaluation.

    By Hill RA, Boland A, Dickson R, Dündar Y, Haycox A, McLeod C, et al.

47. The clinical effectiveness and cost-effectiveness of cardiac resynchronisation (biventricular pacing) for heart failure: systematic review and economic model.

    By Fox M, Mealing S, Anderson R, Dean J, Stein K, Price A, et al.

48. Recruitment to randomised trials: strategies for trial enrolment and participation study. The STEPS study.

    By Campbell MK, Snowdon C, Francis D, Elbourne D, McDonald AM, Knight R, et al.

49. Cost-effectiveness of functional cardiac testing in the diagnosis and management of coronary artery disease: a randomised controlled trial. The CECaT trial.

    By Sharples L, Hughes V, Crean A, Dyer M, Buxton M, Goldsmith K, et al.

50. Evaluation of diagnostic tests when there is no gold standard. A review of methods.

    By Rutjes AWS, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PMM.

51. Systematic reviews of the clinical effectiveness and cost-effectiveness of proton pump inhibitors in acute upper gastrointestinal bleeding.

    By Leontiadis GI, Sreedharan A, Dorward S, Barton P, Delaney B, Howden CW, et al.

52. A review and critique of modelling in prioritising and designing screening programmes.

    By Karnon J, Goyder E, Tappenden P, McPhie S, Towers I, Brazier J, et al.

53. An assessment of the impact of the NHS Health Technology Assessment Programme.

    By Hanney S, Buxton M, Green C, Coulson D, Raftery J.

1. A systematic review and economic model of switching from nonglycopeptide to glycopeptide antibiotic prophylaxis for surgery.

   By Cranny G, Elliott R, Weatherly H, Chambers D, Hawkins N, Myers L, et al.

2. ‘Cut down to quit’ with nicotine replacement therapies in smoking cessation: a systematic review of effectiveness and economic analysis.

   By Wang D, Connock M, Barton P, Fry-Smith A, Aveyard P, Moore D.

3. A systematic review of the effectiveness of strategies for reducing fracture risk in children with juvenile idiopathic arthritis with additional data on long-term risk of fracture and cost of disease management.

   By Thornton J, Ashcroft D, O’Neill T, Elliott R, Adams J, Roberts C, et al.

4. Does befriending by trained lay workers improve psychological well-being and quality of life for carers of people with dementia, and at what cost? A randomised controlled trial.

   By Charlesworth G, Shepstone L, Wilson E, Thalanany M, Mugford M, Poland F.

5. A multi-centre retrospective cohort study comparing the efficacy, safety and cost-effectiveness of hysterectomy and uterine artery embolisation for the treatment of symptomatic uterine fibroids. The HOPEFUL study.

   By Hirst A, Dutton S, Wu O, Briggs A, Edwards C, Waldenmaier L, et al.

6. Methods of prediction and prevention of pre-eclampsia: systematic reviews of accuracy and effectiveness literature with economic modelling.

   By Meads CA, Cnossen JS, Meher S, Juarez-Garcia A, ter Riet G, Duley L, et al.

7. The use of economic evaluations in NHS decision-making: a review and empirical investigation.

   By Williams I, McIver S, Moore D, Bryan S.

8. Stapled haemorrhoidectomy (haemorrhoidopexy) for the treatment of haemorrhoids: a systematic review and economic evaluation.

   By Burch J, Epstein D, Baba-Akbari A, Weatherly H, Fox D, Golder S, et al.

9. The clinical effectiveness of diabetes education models for Type 2 diabetes: a systematic review.

   By Loveman E, Frampton GK, Clegg AJ.

10. Payment to healthcare professionals for patient recruitment to trials: systematic review and qualitative study.

    By Raftery J, Bryant J, Powell J, Kerr C, Hawker S.

11. Cyclooxygenase-2 selective non-steroidal anti-inflammatory drugs (etodolac, meloxicam, celecoxib, rofecoxib, etoricoxib, valdecoxib and lumiracoxib) for osteoarthritis and rheumatoid arthritis: a systematic review and economic evaluation.

    By Chen Y-F, Jobanputra P, Barton P, Bryan S, Fry-Smith A, Harris G, et al.

12. The clinical effectiveness and cost-effectiveness of central venous catheters treated with anti-infective agents in preventing bloodstream infections: a systematic review and economic evaluation.

    By Hockenhull JC, Dwan K, Boland A, Smith G, Bagust A, Dundar Y, et al.

13. Stepped treatment of older adults on laxatives. The STOOL trial.

    By Mihaylov S, Stark C, McColl E, Steen N, Vanoli A, Rubin G, et al.

14. A randomised controlled trial of cognitive behaviour therapy in adolescents with major depression treated by selective serotonin reuptake inhibitors. The ADAPT trial.

    By Goodyer IM, Dubicka B, Wilkinson P, Kelvin R, Roberts C, Byford S, et al.

15. The use of irinotecan, oxaliplatin and raltitrexed for the treatment of advanced colorectal cancer: systematic review and economic evaluation.

    By Hind D, Tappenden P, Tumur I, Eggington E, Sutcliffe P, Ryan A.

16. Ranibizumab and pegaptanib for the treatment of age-related macular degeneration: a systematic review and economic evaluation.

    By Colquitt JL, Jones J, Tan SC, Takeda A, Clegg AJ, Price A.

17. Systematic review of the clinical effectiveness and cost-effectiveness of 64-slice or higher computed tomography angiography as an alternative to invasive coronary angiography in the investigation of coronary artery disease.

    By Mowatt G, Cummins E, Waugh N, Walker S, Cook J, Jia X, et al.

18. Structural neuroimaging in psychosis: a systematic review and economic evaluation.

    By Albon E, Tsourapas A, Frew E, Davenport C, Oyebode F, Bayliss S, et al.

19. Systematic review and economic analysis of the comparative effectiveness of different inhaled corticosteroids and their usage with long-acting beta₂ agonists for the treatment of chronic asthma in adults and children aged 12 years and over.

    By Shepherd J, Rogers G, Anderson R, Main C, Thompson-Coon J, Hartwell D, et al.

20. Systematic review and economic analysis of the comparative effectiveness of different inhaled corticosteroids and their usage with long-acting beta₂ agonists for the treatment of chronic asthma in children under the age of 12 years.

    By Main C, Shepherd J, Anderson R, Rogers G, Thompson-Coon J, Liu Z, et al.

21. Ezetimibe for the treatment of hypercholesterolaemia: a systematic review and economic evaluation.

    By Ara R, Tumur I, Pandor A, Duenas A, Williams R, Wilkinson A, et al.

22. Topical or oral ibuprofen for chronic knee pain in older people. The TOIB study.

    By Underwood M, Ashby D, Carnes D, Castelnuovo E, Cross P, Harding G, et al.

23. A prospective randomised comparison of minor surgery in primary and secondary care. The MiSTIC trial.

    By George S, Pockney P, Primrose J, Smith H, Little P, Kinley H, et al.

24. A review and critical appraisal of measures of therapist–patient interactions in mental health settings.

    By Cahill J, Barkham M, Hardy G, Gilbody S, Richards D, Bower P, et al.

25. The clinical effectiveness and cost-effectiveness of screening programmes for amblyopia and strabismus in children up to the age of 4–5 years: a systematic review and economic evaluation.

    By Carlton J, Karnon J, Czoski-Murray C, Smith KJ, Marr J.

26. A systematic review of the clinical effectiveness and cost-effectiveness and economic modelling of minimal incision total hip replacement approaches in the management of arthritic disease of the hip.

    By de Verteuil R, Imamura M, Zhu S, Glazener C, Fraser C, Munro N, et al.

27. A preliminary model-based assessment of the cost–utility of a screening programme for early age-related macular degeneration.

    By Karnon J, Czoski-Murray C, Smith K, Brand C, Chakravarthy U, Davis S, et al.

28. Intravenous magnesium sulphate and sotalol for prevention of atrial fibrillation after coronary artery bypass surgery: a systematic review and economic evaluation.

    By Shepherd J, Jones J, Frampton GK, Tanajewski L, Turner D, Price A.

29. Absorbent products for urinary/faecal incontinence: a comparative evaluation of key product categories.

    By Fader M, Cottenden A, Getliffe K, Gage H, Clarke-O’Neill S, Jamieson K, et al.

30. A systematic review of repetitive functional task practice with modelling of resource use, costs and effectiveness.

    By French B, Leathley M, Sutton C, McAdam J, Thomas L, Forster A, et al.

31. The effectiveness and cost-effectivness of minimal access surgery amongst people with gastro-oesophageal reflux disease – a UK collaborative study. The reflux trial.

    By Grant A, Wileman S, Ramsay C, Bojke L, Epstein D, Sculpher M, et al.

32. Time to full publication of studies of anti-cancer medicines for breast cancer and the potential for publication bias: a short systematic review.

    By Takeda A, Loveman E, Harris P, Hartwell D, Welch K.

33. Performance of screening tests for child physical abuse in accident and emergency departments.

    By Woodman J, Pitt M, Wentz R, Taylor B, Hodes D, Gilbert RE.

34. Curative catheter ablation in atrial fibrillation and typical atrial flutter: systematic review and economic evaluation.

    By Rodgers M, McKenna C, Palmer S, Chambers D, Van Hout S, Golder S, et al.

35. Systematic review and economic modelling of effectiveness and cost utility of surgical treatments for men with benign prostatic enlargement.

    By Lourenco T, Armstrong N, N’Dow J, Nabi G, Deverill M, Pickard R, et al.

36. Immunoprophylaxis against respiratory syncytial virus (RSV) with palivizumab in children: a systematic review and economic evaluation.

    By Wang D, Cummins C, Bayliss S, Sandercock J, Burls A.

1. Deferasirox for the treatment of iron overload associated with regular blood transfusions (transfusional haemosiderosis) in patients suffering with chronic anaemia: a systematic review and economic evaluation.

   By McLeod C, Fleeman N, Kirkham J, Bagust A, Boland A, Chu P, et al.

2. Thrombophilia testing in people with venous thromboembolism: systematic review and cost-effectiveness analysis.

   By Simpson EL, Stevenson MD, Rawdin A, Papaioannou D.

3. Surgical procedures and non-surgical devices for the management of non-apnoeic snoring: a systematic review of clinical effects and associated treatment costs.

   By Main C, Liu Z, Welch K, Weiner G, Quentin Jones S, Stein K.

4. Continuous positive airway pressure devices for the treatment of obstructive sleep apnoea–hypopnoea syndrome: a systematic review and economic analysis.

   By McDaid C, Griffin S, Weatherly H, Durée K, van der Burgt M, van Hout S, Akers J, et al.

5. Use of classical and novel biomarkers as prognostic risk factors for localised prostate cancer: a systematic review.

   By Sutcliffe P, Hummel S, Simpson E, Young T, Rees A, Wilkinson A, et al.

6. The harmful health effects of recreational ecstasy: a systematic review of observational evidence.

   By Rogers G, Elston J, Garside R, Roome C, Taylor R, Younger P, et al.

7. Systematic review of the clinical effectiveness and cost-effectiveness of oesophageal Doppler monitoring in critically ill and high-risk surgical patients.

   By Mowatt G, Houston G, Hernández R, de Verteuil R, Fraser C, Cuthbertson B, et al.

8. The use of surrogate outcomes in model-based cost-effectiveness analyses: a survey of UK Health Technology Assessment reports.

   By Taylor RS, Elston J.

9. Controlling Hypertension and Hypotension Immediately Post Stroke (CHHIPS) – a randomised controlled trial.

   By Potter J, Mistri A, Brodie F, Chernova J, Wilson E, Jagger C, et al.

10. Routine antenatal anti-D prophylaxis for RhD-negative women: a systematic review and economic evaluation.

    By Pilgrim H, Lloyd-Jones M, Rees A.

11. Amantadine, oseltamivir and zanamivir for the prophylaxis of influenza (including a review of existing guidance no. 67): a systematic review and economic evaluation.

    By Tappenden P, Jackson R, Cooper K, Rees A, Simpson E, Read R, et al.

12. Improving the evaluation of therapeutic interventions in multiple sclerosis: the role of new psychometric methods.

    By Hobart J, Cano S.

13. Treatment of severe ankle sprain: a pragmatic randomised controlled trial comparing the clinical effectiveness and cost-effectiveness of three types of mechanical ankle support with tubular bandage. The CAST trial.

    By Cooke MW, Marsh JL, Clark M, Nakash R, Jarvis RM, Hutton JL, et al. , on behalf of the CAST trial group.

14. Non-occupational postexposure prophylaxis for HIV: a systematic review.

    By Bryant J, Baxter L, Hird S.

15. Blood glucose self-monitoring in type 2 diabetes: a randomised controlled trial.

    By Farmer AJ, Wade AN, French DP, Simon J, Yudkin P, Gray A, et al.

16. How far does screening women for domestic (partner) violence in different health-care settings meet criteria for a screening programme? Systematic reviews of nine UK National Screening Committee criteria.

    By Feder G, Ramsay J, Dunne D, Rose M, Arsene C, Norman R, et al.

17. Spinal cord stimulation for chronic pain of neuropathic or ischaemic origin: systematic review and economic evaluation.

    By Simpson, EL, Duenas A, Holmes MW, Papaioannou D, Chilcott J.

18. The role of magnetic resonance imaging in the identification of suspected acoustic neuroma: a systematic review of clinical and cost-effectiveness and natural history.

    By Fortnum H, O’Neill C, Taylor R, Lenthall R, Nikolopoulos T, Lightfoot G, et al.

19. Dipsticks and diagnostic algorithms in urinary tract infection: development and validation, randomised trial, economic analysis, observational cohort and qualitative study.

    By Little P, Turner S, Rumsby K, Warner G, Moore M, Lowes JA, et al.

20. Systematic review of respite care in the frail elderly.

    By Shaw C, McNamara R, Abrams K, Cannings-John R, Hood K, Longo M, et al.

21. Neuroleptics in the treatment of aggressive challenging behaviour for people with intellectual disabilities: a randomised controlled trial (NACHBID).

    By Tyrer P, Oliver-Africano P, Romeo R, Knapp M, Dickens S, Bouras N, et al.

22. Randomised controlled trial to determine the clinical effectiveness and cost-effectiveness of selective serotonin reuptake inhibitors plus supportive care, versus supportive care alone, for mild to moderate depression with somatic symptoms in primary care: the THREAD (THREshold for AntiDepressant response) study.

    By Kendrick T, Chatwin J, Dowrick C, Tylee A, Morriss R, Peveler R, et al.

23. Diagnostic strategies using DNA testing for hereditary haemochromatosis in at-risk populations: a systematic review and economic evaluation.

    By Bryant J, Cooper K, Picot J, Clegg A, Roderick P, Rosenberg W, et al.

24. Enhanced external counterpulsation for the treatment of stable angina and heart failure: a systematic review and economic analysis.

    By McKenna C, McDaid C, Suekarran S, Hawkins N, Claxton K, Light K, et al.

25. Development of a decision support tool for primary care management of patients with abnormal liver function tests without clinically apparent liver disease: a record-linkage population cohort study and decision analysis (ALFIE).

    By Donnan PT, McLernon D, Dillon JF, Ryder S, Roderick P, Sullivan F, et al.

26. A systematic review of presumed consent systems for deceased organ donation.

    By Rithalia A, McDaid C, Suekarran S, Norman G, Myers L, Sowden A.

27. Paracetamol and ibuprofen for the treatment of fever in children: the PITCH randomised controlled trial.

    By Hay AD, Redmond NM, Costelloe C, Montgomery AA, Fletcher M, Hollinghurst S, et al.

28. A randomised controlled trial to compare minimally invasive glucose monitoring devices with conventional monitoring in the management of insulin-treated diabetes mellitus (MITRE).

    By Newman SP, Cooke D, Casbard A, Walker S, Meredith S, Nunn A, et al.

29. Sensitivity analysis in economic evaluation: an audit of NICE current practice and a review of its use and value in decision-making.

    By Andronis L, Barton P, Bryan S.

30. Trastuzumab for the treatment of primary breast cancer in HER2-positive women: a single technology appraisal.

    By Ward S, Pilgrim H, Hind D.

31. Docetaxel for the adjuvant treatment of early node-positive breast cancer: a single technology appraisal.

    By Chilcott J, Lloyd Jones M, Wilkinson A.

32. The use of paclitaxel in the management of early stage breast cancer.

    By Griffin S, Dunn G, Palmer S, Macfarlane K, Brent S, Dyker A, et al.

33. Rituximab for the first-line treatment of stage III/IV follicular non-Hodgkin’s lymphoma.

    By Dundar Y, Bagust A, Hounsome J, McLeod C, Boland A, Davis H, et al.

34. Bortezomib for the treatment of multiple myeloma patients.

    By Green C, Bryant J, Takeda A, Cooper K, Clegg A, Smith A, et al.

35. Fludarabine phosphate for the firstline treatment of chronic lymphocytic leukaemia.

    By Walker S, Palmer S, Erhorn S, Brent S, Dyker A, Ferrie L, et al.

36. Erlotinib for the treatment of relapsed non-small cell lung cancer.

    By McLeod C, Bagust A, Boland A, Hockenhull J, Dundar Y, Proudlove C, et al.

37. Cetuximab plus radiotherapy for the treatment of locally advanced squamous cell carcinoma of the head and neck.

    By Griffin S, Walker S, Sculpher M, White S, Erhorn S, Brent S, et al.

38. Infliximab for the treatment of adults with psoriasis.

    By Loveman E, Turner D, Hartwell D, Cooper K, Clegg A

39. Psychological interventions for postnatal depression: cluster randomised trial and economic evaluation. The PoNDER trial.

    By Morrell CJ, Warner R, Slade P, Dixon S, Walters S, Paley G, et al.

40. The effect of different treatment durations of clopidogrel in patients with non-ST-segment elevation acute coronary syndromes: a systematic review and value of information analysis.

    By Rogowski R, Burch J, Palmer S, Craigs C, Golder S, Woolacott N.

41. Systematic review and individual patient data meta-analysis of diagnosis of heart failure, with modelling of implications of different diagnostic strategies in primary care.

    By Mant J, Doust J, Roalfe A, Barton P, Cowie MR, Glasziou P, et al.

42. A multicentre randomised controlled trial of the use of continuous positive airway pressure and non-invasive positive pressure ventilation in the early treatment of patients presenting to the emergency department with severe acute cardiogenic pulmonary oedema: the 3CPO trial.

    By Gray AJ, Goodacre S, Newby DE, Masson MA, Sampson F, Dixon S, et al. , on behalf of the 3CPO study investigators.

43. Early high-dose lipid-lowering therapy to avoid cardiac events: a systematic review and economic evaluation.

    By Ara R, Pandor A, Stevens J, Rees A, Rafia R.

44. Adefovir dipivoxil and pegylated interferon alpha for the treatment of chronic hepatitis B: an updated systematic review and economic evaluation.

    By Jones J, Shepherd J, Baxter L, Gospodarevskaya E, Hartwell D, Harris P, et al.

45. Methods to identify postnatal depression in primary care: an integrated evidence synthesis and value of information analysis.

    By Hewitt CE, Gilbody SM, Brealey S, Paulden M, Palmer S, Mann R, et al.

46. A double-blind randomised placebo-controlled trial of topical intranasal corticosteroids in 4- to 11-year-old children with persistent bilateral otitis media with effusion in primary care.

    By Williamson I, Benge S, Barton S, Petrou S, Letley L, Fasey N, et al.

47. The effectiveness and cost-effectiveness of methods of storing donated kidneys from deceased donors: a systematic review and economic model.

    By Bond M, Pitt M, Akoh J, Moxham T, Hoyle M, Anderson R.

48. Rehabilitation of older patients: day hospital compared with rehabilitation at home. A randomised controlled trial.

    By Parker SG, Oliver P, Pennington M, Bond J, Jagger C, Enderby PM, et al.

49. Breastfeeding promotion for infants in neonatal units: a systematic review and economic analysis.

    By Renfrew MJ, Craig D, Dyson L, McCormick F, Rice S, King SE, et al.

50. The clinical effectiveness and cost-effectiveness of bariatric (weight loss) surgery for obesity: a systematic review and economic evaluation.

    By Picot J, Jones J, Colquitt JL, Gospodarevskaya E, Loveman E, Baxter L, et al.

51. Rapid testing for group B streptococcus during labour: a test accuracy study with evaluation of acceptability and cost-effectiveness.

    By Daniels J, Gray J, Pattison H, Roberts T, Edwards E, Milner P, et al.

52. Screening to prevent spontaneous preterm birth: systematic reviews of accuracy and effectiveness literature with economic modelling.

    By Honest H, Forbes CA, Durée KH, Norman G, Duffy SB, Tsourapas A, et al.

53. The effectiveness and cost-effectiveness of cochlear implants for severe to profound deafness in children and adults: a systematic review and economic model.

    By Bond M, Mealing S, Anderson R, Elston J, Weiner G, Taylor RS, et al.

54. Gemcitabine for the treatment of metastatic breast cancer.

    By Jones J, Takeda A, Tan SC, Cooper K, Loveman E, Clegg A.

55. Varenicline in the management of smoking cessation: a single technology appraisal.

    By Hind D, Tappenden P, Peters J, Kenjegalieva K.

56. Alteplase for the treatment of acute ischaemic stroke: a single technology appraisal.

    By Lloyd Jones M, Holmes M.

57. Rituximab for the treatment of rheumatoid arthritis.

    By Bagust A, Boland A, Hockenhull J, Fleeman N, Greenhalgh J, Dundar Y, et al.

58. Omalizumab for the treatment of severe persistent allergic asthma.

    By Jones J, Shepherd J, Hartwell D, Harris P, Cooper K, Takeda A, et al.

59. Rituximab for the treatment of relapsed or refractory stage III or IV follicular non-Hodgkin’s lymphoma.

    By Boland A, Bagust A, Hockenhull J, Davis H, Chu P, Dickson R.

60. Adalimumab for the treatment of psoriasis.

    By Turner D, Picot J, Cooper K, Loveman E.

61. Dabigatran etexilate for the prevention of venous thromboembolism in patients undergoing elective hip and knee surgery: a single technology appraisal.

    By Holmes M, C Carroll C, Papaioannou D.

62. Romiplostim for the treatment of chronic immune or idiopathic thrombocytopenic purpura: a single technology appraisal.

    By Mowatt G, Boachie C, Crowther M, Fraser C, Hernández R, Jia X, et al.

63. Sunitinib for the treatment of gastrointestinal stromal tumours: a critique of the submission from Pfizer.

    By Bond M, Hoyle M, Moxham T, Napier M, Anderson R.

64. Vitamin K to prevent fractures in older women: systematic review and economic evaluation.

    By Stevenson M, Lloyd-Jones M, Papaioannou D.

65. The effects of biofeedback for the treatment of essential hypertension: a systematic review.

    By Greenhalgh J, Dickson R, Dundar Y.

66. A randomised controlled trial of the use of aciclovir and/or prednisolone for the early treatment of Bell’s palsy: the BELLS study.

    By Sullivan FM, Swan IRC, Donnan PT, Morrison JM, Smith BH, McKinstry B, et al.

67. Lapatinib for the treatment of HER2-overexpressing breast cancer.

    By Jones J, Takeda A, Picot J, von Keyserlingk C, Clegg A.

68. Infliximab for the treatment of ulcerative colitis.

    By Hyde C, Bryan S, Juarez-Garcia A, Andronis L, Fry-Smith A.

69. Rimonabant for the treatment of overweight and obese people.

    By Burch J, McKenna C, Palmer S, Norman G, Glanville J, Sculpher M, et al.

70. Telbivudine for the treatment of chronic hepatitis B infection.

    By Hartwell D, Jones J, Harris P, Cooper K.

71. Entecavir for the treatment of chronic hepatitis B infection.

    By Shepherd J, Gospodarevskaya E, Frampton G, Cooper, K.

72. Febuxostat for the treatment of hyperuricaemia in people with gout: a single technology appraisal.

    By Stevenson M, Pandor A.

73. Rivaroxaban for the prevention of venous thromboembolism: a single technology appraisal.

    By Stevenson M, Scope A, Holmes M, Rees A, Kaltenthaler E.

74. Cetuximab for the treatment of recurrent and/or metastatic squamous cell carcinoma of the head and neck.

    By Greenhalgh J, Bagust A, Boland A, Fleeman N, McLeod C, Dundar Y, et al.

75. Mifamurtide for the treatment of osteosarcoma: a single technology appraisal.

    By Pandor A, Fitzgerald P, Stevenson M, Papaioannou D.

76. Ustekinumab for the treatment of moderate to severe psoriasis.

    By Gospodarevskaya E, Picot J, Cooper K, Loveman E, Takeda A.

77. Endovascular stents for abdominal aortic aneurysms: a systematic review and economic model.

    By Chambers D, Epstein D, Walker S, Fayter D, Paton F, Wright K, et al.

78. Clinical and cost-effectiveness of epoprostenol, iloprost, bosentan, sitaxentan and sildenafil for pulmonary arterial hypertension within their licensed indications: a systematic review and economic evaluation.

    By Chen Y-F, Jowett S, Barton P, Malottki K, Hyde C, Gibbs JSR, et al.

79. Cessation of attention deficit hyperactivity disorder drugs in the young (CADDY) – a pharmacoepidemiological and qualitative study.

    By Wong ICK, Asherson P, Bilbow A, Clifford S, Coghill D, R DeSoysa R, et al.

80. ARTISTIC: a randomised trial of human papillomavirus (HPV) testing in primary cervical screening.

    By Kitchener HC, Almonte M, Gilham C, Dowie R, Stoykova B, Sargent A, et al.

81. The clinical effectiveness of glucosamine and chondroitin supplements in slowing or arresting progression of osteoarthritis of the knee: a systematic review and economic evaluation.

    By Black C, Clar C, Henderson R, MacEachern C, McNamee P, Quayyum Z, et al.

82. Randomised preference trial of medical versus surgical termination of pregnancy less than 14 weeks’ gestation (TOPS).

    By Robson SC, Kelly T, Howel D, Deverill M, Hewison J, Lie MLS, et al.

83. Randomised controlled trial of the use of three dressing preparations in the management of chronic ulceration of the foot in diabetes.

    By Jeffcoate WJ, Price PE, Phillips CJ, Game FL, Mudge E, Davies S, et al.

84. VenUS II: a randomised controlled trial of larval therapy in the management of leg ulcers.

    By Dumville JC, Worthy G, Soares MO, Bland JM, Cullum N, Dowson C, et al.

85. A prospective randomised controlled trial and economic modelling of antimicrobial silver dressings versus non-adherent control dressings for venous leg ulcers: the VULCAN trial

    By Michaels JA, Campbell WB, King BM, MacIntyre J, Palfreyman SJ, Shackley P, et al.

86. Communication of carrier status information following universal newborn screening for sickle cell disorders and cystic fibrosis: qualitative study of experience and practice.

    By Kai J, Ulph F, Cullinan T, Qureshi N.

87. Antiviral drugs for the treatment of influenza: a systematic review and economic evaluation.

    By Burch J, Paulden M, Conti S, Stock C, Corbett M, Welton NJ, et al.

88. Development of a toolkit and glossary to aid in the adaptation of health technology assessment (HTA) reports for use in different contexts.

    By Chase D, Rosten C, Turner S, Hicks N, Milne R.

89. Colour vision testing for diabetic retinopathy: a systematic review of diagnostic accuracy and economic evaluation.

    By Rodgers M, Hodges R, Hawkins J, Hollingworth W, Duffy S, McKibbin M, et al.

90. Systematic review of the effectiveness and cost-effectiveness of weight management schemes for the under fives: a short report.

    By Bond M, Wyatt K, Lloyd J, Welch K, Taylor R.

91. Are adverse effects incorporated in economic models? An initial review of current practice.

    By Craig D, McDaid C, Fonseca T, Stock C, Duffy S, Woolacott N.

1. Multicentre randomised controlled trial examining the cost-effectiveness of contrast-enhanced high field magnetic resonance imaging in women with primary breast cancer scheduled for wide local excision (COMICE).

   By Turnbull LW, Brown SR, Olivier C, Harvey I, Brown J, Drew P, et al.

2. Bevacizumab, sorafenib tosylate, sunitinib and temsirolimus for renal cell carcinoma: a systematic review and economic evaluation.

   By Thompson Coon J, Hoyle M, Green C, Liu Z, Welch K, Moxham T, et al.

3. The clinical effectiveness and cost-effectiveness of testing for cytochrome P450 polymorphisms in patients with schizophrenia treated with antipsychotics: a systematic review and economic evaluation.

   By Fleeman N, McLeod C, Bagust A, Beale S, Boland A, Dundar Y, et al.

4. Systematic review of the clinical effectiveness and cost-effectiveness of photodynamic diagnosis and urine biomarkers (FISH, ImmunoCyt, NMP22) and cytology for the detection and follow-up of bladder cancer.

   By Mowatt G, Zhu S, Kilonzo M, Boachie C, Fraser C, Griffiths TRL, et al.

5. Effectiveness and cost-effectiveness of arthroscopic lavage in the treatment of osteoarthritis of the knee: a mixed methods study of the feasibility of conducting a surgical placebo-controlled trial (the KORAL study).

   By Campbell MK, Skea ZC, Sutherland AG, Cuthbertson BH, Entwistle VA, McDonald AM, et al.

6. A randomised 2 × 2 trial of community versus hospital pulmonary rehabilitation for chronic obstructive pulmonary disease followed by telephone or conventional follow-up.

   By Waterhouse JC, Walters SJ, Oluboyede Y, Lawson RA.

7. The effectiveness and cost-effectiveness of behavioural interventions for the prevention of sexually transmitted infections in young people aged 13–19: a systematic review and economic evaluation.

   By Shepherd J, Kavanagh J, Picot J, Cooper K, Harden A, Barnett-Page E, et al.

8. Dissemination and publication of research findings: an updated review of related biases.

   By Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al.

9. The effectiveness and cost-effectiveness of biomarkers for the prioritisation of patients awaiting coronary revascularisation: a systematic review and decision model.

   By Hemingway H, Henriksson M, Chen R, Damant J, Fitzpatrick N, Abrams K, et al.

10. Comparison of case note review methods for evaluating quality and safety in health care.

    By Hutchinson A, Coster JE, Cooper KL, McIntosh A, Walters SJ, Bath PA, et al.

11. Clinical effectiveness and cost-effectiveness of continuous subcutaneous insulin infusion for diabetes: systematic review and economic evaluation.

    By Cummins E, Royle P, Snaith A, Greene A, Robertson L, McIntyre L, et al.

12. Self-monitoring of blood glucose in type 2 diabetes: systematic review.

    By Clar C, Barnard K, Cummins E, Royle P, Waugh N.

13. North of England and Scotland Study of Tonsillectomy and Adeno-tonsillectomy in Children (NESSTAC): a pragmatic randomised controlled trial with a parallel non-randomised preference study.

    By Lock C, Wilson J, Steen N, Eccles M, Mason H, Carrie S, et al.

14. Multicentre randomised controlled trial of the clinical and cost-effectiveness of a bypass-surgery-first versus a balloon-angioplasty-first revascularisation strategy for severe limb ischaemia due to infrainguinal disease. The Bypass versus Angioplasty in Severe Ischaemia of the Leg (BASIL) trial.

    By Bradbury AW, Adam DJ, Bell J, Forbes JF, Fowkes FGR, Gillespie I, et al.

15. A randomised controlled multicentre trial of treatments for adolescent anorexia nervosa including assessment of cost-effectiveness and patient acceptability – the TOuCAN trial.

    By Gowers SG, Clark AF, Roberts C, Byford S, Barrett B, Griffiths A, et al.

16. Randomised controlled trials for policy interventions: a review of reviews and meta-regression.

    By Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, et al.

17. Paracetamol and selective and non-selective non-steroidal anti-inflammatory drugs (NSAIDs) for the reduction of morphine-related side effects after major surgery: a systematic review.

    By McDaid C, Maund E, Rice S, Wright K, Jenkins B, Woolacott N.

18. A systematic review of outcome measures used in forensic mental health research with consensus panel opinion.

    By Fitzpatrick R, Chambers J, Burns T, Doll H, Fazel S, Jenkinson C, et al.

19. The clinical effectiveness and cost-effectiveness of topotecan for small cell lung cancer: a systematic review and economic evaluation.

    By Loveman E, Jones J, Hartwell D, Bird A, Harris P, Welch K, et al.

20. Antenatal screening for haemoglobinopathies in primary care: a cohort study and cluster randomised trial to inform a simulation model. The Screening for Haemoglobinopathies in First Trimester (SHIFT) trial.

    By Dormandy E, Bryan S, Gulliford MC, Roberts T, Ades T, Calnan M, et al.

21. Early referral strategies for management of people with markers of renal disease: a systematic review of the evidence of clinical effectiveness, cost-effectiveness and economic analysis.

    By Black C, Sharma P, Scotland G, McCullough K, McGurn D, Robertson L, et al.

22. A randomised controlled trial of cognitive behaviour therapy and motivational interviewing for people with Type 1 diabetes mellitus with persistent sub-optimal glycaemic control: A Diabetes and Psychological Therapies (ADaPT) study.

    By Ismail K, Maissi E, Thomas S, Chalder T, Schmidt U, Bartlett J, et al.

23. A randomised controlled equivalence trial to determine the effectiveness and cost–utility of manual chest physiotherapy techniques in the management of exacerbations of chronic obstructive pulmonary disease (MATREX).

    By Cross J, Elender F, Barton G, Clark A, Shepstone L, Blyth A, et al.

24. A systematic review and economic evaluation of the clinical effectiveness and cost-effectiveness of aldosterone antagonists for postmyocardial infarction heart failure.

    By McKenna C, Burch J, Suekarran S, Walker S, Bakhai A, Witte K, et al.

Health Technology Assessment programme

1. Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

2. Director, Medical Care Research Unit, University of Sheffield