Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems

Bayesian methods

Bayes’ theorem describes how the probability that an individual has a disease (known as the pre-test or prior probability) changes when the result of a diagnostic test is obtained (post-test probability), dependent on the performance characteristics of the test. Bayes’ theorem can be extended to combine multiple pieces of diagnostic information, with the post-test probability obtained from the first test acting as the prior probability for the next test. However, this approach, known as naive Bayes, has been shown to give over-optimistic predictions when individual test results are not independent due to the double counting of diagnostic information.

Logistic regression extensions of Bayes’ theorem

The problem of double counting of diagnostic information is removed by using logistic regression models that account for correlations between the different pieces of diagnostic information. In this method the links between Bayes’ theorem and logistic regression models can be fitted by re-expressing the theorem using ‘weights of evidence’ or log likelihood ratios to account for the correlations. 73 Adjustments for correlations between diagnostic items are made by estimating a beta parameter for each test, which either increases or decreases the likelihood ratio for the test. Bivariate and multivariate model fitting approaches can then be used to remove redundant symptoms and select those to keep in the final model.

Discrimination rules

Discrimination rules use standard statistical methods to produce a rule that can be used to discriminate between individuals on the basis of symptoms or test results, and to allocate them to the group to which they are most likely to belong, for example, diseased versus non-diseased. These methods rely on producing predictions from either logistic regression models or discriminant functional analyses to predict membership of two or more groups. The application of logistical regression in this instance differs in two important ways from that applied to Bayes’ theorem. Firstly, there is no direct way of altering predictions to allow for differences in pre-test probabilities, and secondly, there is no quantification of the uncertainty around group allocation.

Clinical algorithms

An algorithm is a process for carrying out a complex task which is broken down into a series of simple decision and action steps. 74 Clinical algorithms can either be represented as paper-based flowcharts or as computer programs. These algorithms have a number of limitations including the need to have all the data specified in the algorithm available, space restrictions if the algorithm is paper based, and the practical difficulties of breaking down many clinical problems into a set of discrete decisions that can then be represented in computer language. Optimal clinical algorithms can be developed using statistical methods known as classification and regression trees.

Expert systems

An expert system is a computer program that simulates human thought processes ‘to provide the kind of problem analysis and advice that the expert might provide’. 74

Machine learning

Machine learning can either be supervised or unsupervised. In supervised learning the system is provided with a sample of input data and the content designated on how to identify and classify patterns within the data. In unsupervised learning the system is provided with data, but is left to identify patterns without external assistance using a form of cluster analysis. There are a number of different types of machine learning methods, including decision trees, artificial neural networks and genetic algorithms. For all methods, internal weights within the system are adjusted during training until a pre-specified performance level is attained. Limitations of these systems include the fact that although experts can evaluate a decision tree generated by a machine system, the system cannot often provide understandable reasons for the advice it generates. Furthermore experts can rarely evaluate the reasoning behind the classifiers generated by neural networks and genetic algorithms as these systems are ‘black boxes’.

Study question two:

Studies which report an objective measure of process of care, e.g. test volumes, compliance with guidelines implemented via the CDSS, appropriateness of the test(s) ordered, patient outcomes, or adverse events, are included. Studies which only report the diagnostic accuracy of the CDSS compared to a gold standard (such as a diagnosis reached by the clinician without use of the CDSS) (i.e. sensitivity and specificity) are excluded. These studies are excluded as the outcome of interest is the impact of CDSS in conjunction with OCS for test ordering, rather than the accuracy of CDSS compared with a clinician in providing a correct clinical diagnosis.

Study question three:

Outcomes that were included were clinician or patient self-reported acceptability of the CDSS. Acceptability was defined according to the definitions used in the primary studies.

Study question 4:

Studies which reported the cost-effectiveness of CDSS are included.

Identification of relevant studies

A generic search to identify potentially relevant studies for inclusion in the three reviews was conducted. This was used to identify relevant clinical, cost-effectiveness and cost–comparison studies indexed on the following medical and social science databases between 1974 (the year of publication of the first article to evaluate the effect of a CDSS on clinician performance by De Dombal and colleagues)80 and 2008: The searches were then updated in April 2009.

• MEDLINE

• EMBASE

• Cochrane Controlled Trials Register (CCTR)

• CINAHL (Cumulative Index to Nursing and Allied Health Literature)

• DARE (Database of Abstracts of Reviews of Effects)

• Health Technology Assessment (HTA) database

• IEEE (Institute of Electrical and Electronic Engineers) Xplore

• NHS Economic Evaluation Database (NHS EED)

• EconLit.

No study design filters were applied to the search strategy.

The literature searches retrieved 22,109 unique references after de-duplication. All references were managed using reference manager, software version 11. Full details of the search strategies are presented in Appendix 1. In addition, bibliographies of all included studies were checked to identify further relevant studies.

Selection of primary studies for the reviews

Relevant studies were identified in two stages. One reviewer screened titles and abstracts returned by the database searches, and a random 20% of these were checked for agreement by a second reviewer. Cohen’s unweighted κ-statistic for disagreements at the title/abstract screening stage between reviewers was 0.89, indicating a good level of agreement. The full texts of any references that were considered relevant by either reviewer were obtained where available. The relevance of each paper was assessed according to the criteria set out below for each review question. Any discrepancies between the reviewers were resolved by recourse to the papers, and if necessary a third reviewer was consulted. All duplicate papers were double-checked and excluded. The extent of disagreements between reviewers for study inclusion in the reviews was again quantified using Cohen’s unweighted κ-statistic with an agreement level of 0.91 between reviewers. 81 Further bibliographic details of excluded studies, along with reasons for their exclusion are detailed in Appendix 2.

Data extraction and quality assessment processes

Data were extracted from the included studies using a standardised data extraction form developed for each of the reviews. The quality of the individual studies was assessed according to study design by one reviewer and checked for accuracy by a second. RCTs, cluster randomised controlled trials (CRCTs), controlled clinical trials (CCTs), and CPPs and UPPs were assessed according to methodological criteria listed in the up-dated CRD Report 4;79 ITS studies were assessed according to criteria specified by the Cochrane Effective Practice and Organisation of Care (EPOC) Group;82 economic evaluations were assessed using the Consensus on Health Economic Criteria list questions developed by Evers and colleagues. 83

The main criteria assessed according to study design are outlined below.

Inclusion and exclusion criteria

The inclusion and exclusion criteria to select studies for the review on the impact of CDSS in OCS on process and patient outcomes were the same as those outlined in Chapter 2, for the participants, interventions, relevant comparators, outcomes and study designs.

Data extraction strategy

Data were extracted on the study setting, clinician and patient characteristics (where reported), study design and methods, intervention and comparator systems, area of impact, CDSS characteristics, including the presence or absence of the 14 relevant features of CDSS identified as being potentially related to system success by Kawamoto and colleagues,16 and outcomes. Outcomes were summarised using descriptive summary statistics, including proportions (with 95% CIs) for categorical variables and mean [standard deviation (SD)] for continuous variables.

Data synthesis

Due to considerable heterogeneity between the studies, in terms of the type of CDSS evaluated, the output format of the CDSS, study designs and setting, and outcomes assessed, results were first broadly grouped according to the intervention (i.e. the type of information provided by the CDSS, for example, the display of test costs, corollary orders, or advice/recommendations). Studies within each broad intervention group were then further grouped according to study design. Due to the heterogeneity between studies results were therefore combined using a narrative synthesis84,85 with key demographic data for each study and relevant quantitative results tabulated. All data extraction tables are presented in Appendix 3.

Inclusion and exclusion criteria

Inclusion criteria for eligible studies were the same as those specified in Chapter 2, for study participants. Inclusion criteria for the interventions were also the same as those listed in Chapter 2, apart from a comparator system, i.e. an OCS without CDSS was not required for studies to be eligible for inclusion. The study outcomes that were included were clinician or patient self-reported acceptability of the CDSS. Acceptability was defined according to the definitions used in the primary studies. Study designs that were included were the same as those used to address review question 2 (i.e. RCTs, CRCTs, CCTs, ITS, CPPs and UPPs) but in addition cross sectional and longitudinal surveys, and qualitative studies were also eligible for inclusion.

Data extraction strategy

The data extracted included the study setting; clinician and patient characteristics; study methods; intervention and comparator systems (where applicable); self-reported rates/scores of clinician or patient acceptability of the CDSS; and CDSS characteristics including where reported, time taken to obtain the CDSS, time taken to use the CDSS, methods of system reasoning, CDSS knowledge base, number of data items to collect (if the data are not already available in electronic patient records), ease of data entry, ease of interpreting results and perceived applicability of the CDSS knowledge base to the clinician’s own patients. All data on clinician or patient acceptance of the CDSS were summarised using appropriate descriptive summary measures including proportions (with 95% CIs) for categorical variables.

Data synthesis

Due to considerable heterogeneity between the studies, in terms of the type of CDSS evaluated, the output format of the CDSS, study designs and setting, and outcomes assessed, results were first broadly grouped according to the intervention (i.e. the type of information provided by the CDSS, for example, the display of test costs, recommendations, or restricted lists). Where studies incorporated a second subsidiary CDSS intervention the study was grouped according to the primary outcome and aim of evaluating the CDSS, with the secondary outcome also reported within this category. Studies within each broad intervention group were then further subgrouped according to study design. Due to the heterogeneity between studies results were combined using a narrative synthesis84,85 with key demographic data for each study and relevant quantitative results tabulated.

Studies were grouped according to the type of CDSS system with key data on acceptability and specific system features presented in tables. Results were then combined using a narrative synthesis. 84,85 Differences in rates/scores of acceptability between studies were explored narratively by recourse to differences in the study setting, and CDSS characteristics.

Inclusion and exclusion criteria

The inclusion and exclusion criteria for the systematic review of economic evaluations and cost–comparison studies were identical to those specified in Chapter 2, apart from the study design criteria. For the review, full CEA, CUA, cost–benefit analyses (CBA), CCA, and cost–comparison studies were included. Economic evaluations that only reported average cost-effectiveness ratios were only eligible for inclusion if incremental ratios could be calculated from the available published data.

Data extraction strategy

Data on study setting, clinician and patient characteristics (where reported), intervention, comparators, and outcomes were tabulated. In addition data were extracted on the study design (CEA, CUA or cost-analysis), model type or trial based study, research question, perspective, time horizon, discounting, main costs included, and sensitivity analyses.

Methods of data synthesis

Studies results were presented narratively and where possible key results presented in tables. Differences in the cost-effectiveness of CDSS in comparison with OCS alone were explored narratively by considering differences in the setting, type of tests ordered, type of CDSS and OCS, perspective, time horizon and methods of discounting.

Overview of the quantity and quality of the research available

A total of 24 studies reported in 23 publications met the inclusion criteria. 29,32,48,50,57,86–101 The results from two RCTs, one assessing the impact of the display of test charges on clinical laboratory test orders and the other the impact on radiological test volumes conducted at the same institution, were reported together in one paper by Bates and colleagues. 101 Two further studies included in the review of the impact of CDSS in OCS also reported limited cost–comparison data and were therefore also included in the review of the cost-effectiveness of CDSS in OCS compared to OCS alone. 96,98

As previously stated there was considerable heterogeneity between the identified studies in terms of the type of CDSS assessed, the settings in which the studies were conducted, the patient populations, whether the studies focused on the impact of the CDSS on a single type of laboratory or imaging test order or on multiple tests and study designs. The nomenclature used to attempt to group the studies is therefore highly simplistic, as a limited number of studies assessed the impact of multiple interventions. However, in grouping the studies the predominant CDSS intervention, for example the display of restricted test order lists, was chosen for the grouping. This means that the results of these studies are more likely to be confounded from concomitant interventions than is readily apparent from rudimentary methods used to devise the study groupings.

Overall, in total, the 24 studies could broadly be grouped into those that assessed the impact of the display of test charges (n = 3),86,101 those that displayed patients previous test results (n = 2),88,89 those that provided reminders (n = 10),24,29,48,87,91–94,102,103 studies that displayed restricted lists of test orders (n = 2),95,96 and those in which the CDSS provided a recommendation (n = 7). 29,32,48,50,57,86–101 A summary of the type of CDSS intervention by the number of studies and study design is displayed in Table 4.

Across the 24 studies, seven CRCTs (29%),24,32,48,50,86,90,91 four RCTs (17%),92,97,101 two non-randomised controlled trials (8%),57,88 one randomised crossover trial (4%),29 two ITS studies (one with a AB-AB-AB design) (8%),93,95 one CPP study (4%),96 and seven UPP studies (29%)87,89,94,98–100,104 were identified. Duration of follow-up varied widely with a median of 7 months (range: 2–72). Sixty-five per cent of studies described funding from the public sector,24,29,48,86,89–92,94–97,101 13% stated funding was from the private sector,50,57,98 while 22% did not report the funding source. 32,87,93,99,100 Developers of the CDSS software were also outcome evaluators in 62.5% of the studies,24,29,48,86,88,89,91–93,95,96,99,101,104 were evaluators in part (collaboration) in one study (8%),90 and were not involved in the system evaluation in 29% of the studies. 50,57,87,94,97,98,105

In terms of the study settings, of the 24 studies the majority (17, 71%) were conducted in the USA,23,28,47,86–91,94–96,98,103 followed by two (8%) each conducted in the UK32,99 and Spain. 7,98 The remaining three studies were conducted in France (4%),93 the Netherlands (4%),96 and Belgium (4%) respectively. 100 Of the studies conducted in the USA, 12 of the 17 studies had been undertaken at one of three specific sites: four studies were conducted at the Wishard Memorial Hospital, Indianapolis, IN, or at one or their outpatient centres;24,29,48,86 five had been conducted at the Brigham and Women’s Hospital, Boston, MA;88,91,92,101 and three had been conducted at the Vanderbilt University Medical Center. 89,95,104 Across the 24 studies, eight were conducted in a primary care setting,32,50,57,86,90,94,96,98 one was conducted in both a primary and secondary outpatient care setting,91 two were conducted in secondary care outpatients,29,97 11 were conducted in secondary care inpatients,24,48,87,88,92,95,99–101,104 one was conducted in a secondary care intensive care unit (ICU),89 and one in an accident and emergency department. 93

In relation to the patients indication(s) for undergoing either laboratory or radiological test imaging, the majority of the studies (n = 10) included patients with a mixture of diagnoses that were unspecified. 24,29,48,86,87,90–92,94,97 Two studies included only medical or surgical patients,101 and one included patients undergoing testing for suspected rheumatic disease. 88 The focus of a further three studies were test specific, and so included patients undergoing arterial blood gas (ABG) laboratory testing,89 serum magnesium level testing,95 or a range of blood tests96 respectively. Of the remaining studies, one included patients with a diagnosis of diabetes mellitus,57 three included patients with hyperlipidemia,32,50,98 two included patients undergoing assessment or liver transplantation,101,102 and two included patients in which radiological imaging was indicated. 93,104

The outcomes reported reflected the intended impact of the CDSS and included test volumes,32,50,86,88,89,95,96,98,99,101,104,105 test costs,86,97,99,101, compliance with reminders,24,29,48,90,91,94 compliance with recommendations,97,104 guideline compliance,87,93 and order appropriateness in terms of test frequency. 57,92 Only four studies reported potential adverse effects of test cancellations. 86,92,101 Additionally, all the studies focused on the decision to order a laboratory or imaging test, rather than on the impact of CDSS on the interpretation of test results. Of the 24 included studies, six were focused more broadly on patient management, and included the ordering or appropriateness of pharmacological prescriptions, vaccinations or health-care advice. Therefore only limited outcomes in terms of the impact of CDSS on laboratory test rates or appropriateness were reported in these studies. 24,29,48,50,97,98

As well as the studies being heterogeneous in terms of the type of CDSS assessed, the settings, patient groups and study designs, the year in which they were undertaken and published also varied considerably. Of the 24 eligible studies, one was published in the period 1980–4, two in 1990–4, eight in 1995–9, three in 2000–4 and 10 in 2004–9.

While the year of study publication can only act as a proxy for the year(s) in which the research was conducted, it can be postulated, particularly in terms of the older CDSS, that these systems may now be obsolete, or will have been upgraded and changed considerably with further technological development in clinical settings. Many of the older studies may therefore now not be of direct relevance to CDSS that are currently available, and greater weight should be given to the more recently published studies that have assessed technologies that are still available or may have undergone limited changes. A summary of the number of studies identified by year is displayed in Figure 4 and a summary of study characteristics of the 24 identified studies in Table 5.

FIGURE 4.

Number of included studies by year.

Study quality

As can be expected from the heterogeneity of the identified studies, the level of reporting and study quality was highly variable. Across the studies the median length of follow-up was 7 months, but this ranged dramatically, from 2 months to 72 months. An assessment of study quality is provided in each specific section of the report according to CDSS intervention type.

Overview of the CDSS characteristics according the 15 features of CDSS suggested by Kawamoto and colleagues16 as having a potential impact on CDSS effectiveness

An overview of the CDSS characteristics from the 24 studies postulated by Kawamoto and colleagues16 as having a potential impact on the effectiveness of the CDSS is displayed in Figure 5. The figure depicts 14 of the characteristics, but omits the use of a computer to generate the decision support, as this formed part of the inclusion criteria by which studies were selected for inclusion in the review, and therefore is not relevant to the current review scope.

FIGURE 5.

Overview of the CDSS characteristics of the included studies. NA, not applicable; NR, not reported.

As can be seen from Figure 5 all the studies were integrated with charting or OCS to support workflow integration, provided automatic decision support as part of the clinician workflow, and provided support at the time and location of decision-making. In the majority of studies (79%, 19/24) there was no need for additional data entry by the clinician. Again in the majority of studies (67%, 16/24) the CDSS did not request documentation for not following the recommendations (where these were made), and in 83% (20/24) of studies there was no need for the clinician to execute any recommendations by noting agreement.

Only 38% (9/24) of studies provided a recommendation with the rest displaying either test charges, previous test results, reminders, or displaying restricted test ordering lists. In the majority of studies (79%, 19/24) it was not reported whether the CDSS output would be likely to promote action by the clinician rather than inaction (for example, suggesting a different course of action or test if appropriate rather than suggesting that the test order was cancelled). Additionally in 88% (21/24) of studies no justification of the CDSS output was provided either by recourse to the provision of CDSS reasoning or the research evidence on which this was based.

Local users of the CDSS were involved in the development process in only 17% (4/24) of the studies, with the majority of studies (63%, 15/24) not involving the health-care professionals who would ultimately use the system either in the development or piloting of the system. None of the studies provided decision support results to patients as well as clinicians, and only 8% (2/24) of the studies provided periodic performance feedback to clinicians. Additionally, only 4% (1/24) of the studies provided concomitant conventional education alongside use of the CDSS.

Quantity and quality of the studies

The impact of the display of test charges on the number of laboratory and radiology test orders was assessed in one CRCT and two RCTs reported in two publications. 86,101 In all three trials, the objective was to reduce the number of tests that were ordered. The CRCT, by Tierney and colleagues86 was conducted in the outpatient General Medicine Practice of the Regenstrief Health Centre, Indianapolis, IN, USA (the primary outpatient facility for the Wizard Memorial Hospital). In the CRCT the physician was the unit of randomisation, with 121 physicians with a total of 8392 patient visits during the intervention period included. 86 The trial consisted of a 14-week pre-intervention period, a 26-week intervention, and a 26-week post-intervention period conducted simultaneously from January 1988. The trial included all patients and visits to physicians, whether scheduled or unscheduled. It also included all outpatient diagnostic tests (i.e. all laboratory and radiological imaging studies) but excluded 24-hour electrocardiographic monitoring, treadmill exercise testing, and endoscopic procedures. 86 The specific CDSS evaluated was not reported, but as the primary care practice was associated with the Wishard Memorial Hospital, it can be postulated to be part of the ‘home-grown/in-house’ OCS and CDSS developed specifically for use at this site. Physicians in both the intervention and control groups entered all their orders for tests through a computer workstation. The CDSS intervention in the trial consisted of the display of the charge the patient (or the insurer) would pay for the current test when ordered and the total charges for all tests ordered for that patient on that day. Additional fees (e.g. for the interpretation of test results) were not included.

The two simultaneously conducted RCTs by Bates and colleagues101 were undertaken on all medical and surgical inpatients at Brigham and Women’s Hospital, Boston, MA, USA. The trials consisted of an assessment of the impact of the display of test charges on laboratory and radiological imaging test orders (length of trial follow-up 4 and 7 months respectively). The patient was the unit of randomisation in the trials, with 7090 patients included in the laboratory trial, and 17,381 in the radiology trial. Therefore physicians could treat both intervention and control group patients, therefore contamination was a risk thereby potentially reducing the effect size of any potential benefits of the CDSS. The trials were conducted between February and October 1994. The trials included all laboratory test orders and the 35 most frequently ordered imaging tests. Charges for the remainder of the radiological tests were not displayed. Physicians could enter test orders for patients though the OCS, with individual and total test charges for each patient order displayed to physicians in the intervention group. Alternatively during the trial periods, specimens could be obtained and sent directly to the laboratory. Therefore the number of tests ordered during the trials was less than the number performed. For the laboratory and radiological trials respectively only 53% and 74% of tests performed had a corresponding computer order. Again, the specific CDSS assessed was not reported, but would appear to be part of the home-grown/in-house OCS and CDSS developed by Brigham and Women’s Hospital. A summary of the study characteristics from the trials is displayed in Table 6.

Outcomes

Study quality

The methods of randomisation and whether allocation concealment was attained was not reported in any of the three trials. 86,101 Trial eligibility criteria were adequately specified and sufficient details of physician/patient baseline characteristics were reported. In all three trials baseline characteristics were balanced between treatment groups, indicating that the method of randomisation, while not specified, was probably appropriate. Blinding of physicians as in all trials of CDSS plus OCS to treatment allocation was not possible, and it was unclear whether outcome assessors were blinded. Moreover, as the unit of randomisation was the patient in the two trials by Bates and colleagues101 contamination bias may be present. This could potentially lead to an underestimation of the impact of the CDSS. Data analyses to account for clustering by physician in the trial by Tierney and colleagues86 was adequate and appropriate. Data analyses were only conducted on an ‘intention to provide or communicate information’ basis in the two trials by Bates and colleagues. 101 However, the rate of attrition (3.2%) was low in the trial by Tierney and colleagues86 and therefore failure to conduct the analysis on an ‘intention to provide or communicate information’ basis is unlikely to impact significantly on the results attained.

CDSS characteristics

A summary of the key characteristics, including the 14 features of CDSS proposed by Kawamoto and colleagues16 as predictors of system success or failure are displayed in Table 7. The specific CDSS, as stated in Quantity and quality of the studies was not reported in any of the trials, but in the trial by Tierney and colleagues86 would appear to be the home-grown/in-house OCS and CDSS developed by the Wishard Memorial Hospital, Indianapolis, IN, USA. In the two trials by Bates and colleagues101 the CDSS again appears to be a home-grown/in-house system, this time developed specifically by Brigham and Women’s Hospital, Boston, MA, USA for use at this site.

The CDSS reasoning methods were not reported in any of the three trials. 86,101 Both systems used laboratory and radiological test costs as the system information source, and the display of test costs as the output format. 86,101 The time to complete the CDSS was only reported in the trial by Tierney and colleagues86 with physicians in the intervention group taking an average of 11.5 seconds for test ordering, compared with 10 seconds taken by physicians in the control group. None of the trials describe any form of user training prior to implementation of the CDSS. 86,101

In relation to the 14 features of CDSS proposed by Kawamoto and colleagues16 the CDSS in all three trials was integrated as part of the OCS, did not require additional data entry by the physician, and provided output automatically as part of the consultation workflow. 86,101 No reasons were required to be documented by the system for not following CDSS recommendations, as only test and total test charges were displayed, and therefore no specific recommendations to order or cancel specific tests were provided by the system. Again, as only test charges were displayed rather than specific recommendations, there was no need for these to be justified by recourse to either the system reasoning methods or the provision of research evidence supporting the recommendations.

It was unclear whether local system users were directly involved in the CDSS development. The data provided by the CDSS was used by the physician alone, and did not appear to provide period summaries of performance feedback. In all three trials the CDSS output information appeared to be used alone, and was not combined in conjunction with educational information regarding the charges for specific tests, and the indications for test ordering. 86,101

Results

Summary of studies assessing the impact of the display of test charges

Quantity and quality of the studies

Two studies, one CCT and one UPP study assessed the impact of the display of previous test results on subsequent test order volumes. 88,89

The CCT conducted by Solomon and colleagues88 was carried out at the Brigham and Women’s Hospital, Boston, MA, USA, and was focused on reducing unnecessary serological testing in the diagnosis of suspected systemic rheumatic disease. 88 The trial was conducted over a 10-month period, with the intervention and control groups formed according to the indication for test(s) ordered. All physicians ordering a rheumatoid factor (RF) or antinuclear antibody (ANA) test for the suspected indications of rheumatoid arthritis, systemic lupus erythematosus, primary systemic sclerosis, mixed connective tissue disease, or Sjögren’s syndrome were assigned to the intervention group. All test orders for RF or ANA for the suspected indications of systemic vasculitis and cryoglobulinemia, or a complement test for any condition during the study period were assigned to the control group. Therefore, during the trial physicians were exposed to both the intervention and control conditions. Tests were selected as target tests for the trial on the basis that estimates of the test’s sensitivity and specificity for each of the suspected indications existed in the literature. The CDSS intervention required physicians to state their estimate of the pre-test probability of disease in the patient. The CDSS then calculated the post-test positive and negative predictive values, based on the physician’s estimated pre-test probability. These calculations were based on sensitivity and specificity values abstracted from relevant literature. 106–112 During the 10-month trial 71 physicians wrote test orders for 99 patients in the intervention group, while 154 physicians wrote orders for 236 patients in the control group. The two groups were well balanced in terms of both physician (age, gender, postgraduate year, department) and patient (age, gender, length of hospital stay, total hospital charges) baseline characteristics.

The UPP study conducted by Bansal and colleagues89 aimed to assess the impact of a computer-based intervention on ABG usage in an ICU setting. The study was conducted at Vanderbilt University Medical Centre, Nashville, TN, USA, and included six conditions managed in ICU (trauma, general surgery, medical, cardiac, burn and neurology). The study was conducted over a 12-week period, consisting of a 5-week pre-intervention and a 7-week intervention period. There were no restrictions on the ordering of ABG tests during the pre-intervention period (OCS alone).

During the intervention the CDSS provided the user with a graphical display of the patient’s previous ABG values; pO₂ pCO₂, HCO₃, O₂ saturation, pH, and FiO₂. All values, except O₂ saturations, reflected previous ABGs performed during the patient’s current hospitalisation. In addition cointervention educational text was provided alongside the graphical display of results and test ordering was limited to within 24 hours so no multi-day orders were allowed. The default (pre-populated) response was to cancel the order. However, the final decision to test or not was left to the user’s discretion. Table 10 displays a summary of the key characteristics of both studies.

Quantity and quality of the studies

The impact of the display of reminders was assessed in 10 studies:24,29,48,57,87,90–94

• four CRCTs, two published by Overhage and colleagues in 1996 and 1997,24,48 one by Palen and colleagues,90 one by Matheny and colleagues91

• one RCT undertaken by Bates and colleagues92

• one CCT by O’Connor and colleagues57

• one randomised crossover trial by McDonald and colleagues29

• one ITS study with an AB-AB-AB design by Carton and colleagues93 and

• two UPP studies by Steele and colleagues and Abboud and colleagues. 87,94

Across the studies, nine of the 10 were conducted in the USA,24,29,48,57,87,90–92,94 while the remaining study was conducted in France. 93 Of those undertaken in the USA, three studies were undertaken at the Wishard Memorial Hospital, Indianapolis, IN,24,29,48 two of which were undertaken in an inpatient setting24,48 and one in an outpatient setting. 29 Two studies were conducted at the Brigham and Women’s Hospital, and associated community and outpatient clinics,91,92 one of which included inpatients and the other outpatients. Of the remaining four studies undertaken in the USA, three were undertaken in an outpatient setting (group-model managed care organisation, Kaiser Permanente; Health Partners Medical Group, MN; and Sam Sandos Family Health Clinic, Denver Health respectively)57,90,94 and one was undertaken in a paediatric inpatient population at Cincinnati Children’s Hospital Medical Centre, OH. 87 Of these nine studies, one assessed compliance with reminders to undertake preventative care measures,48 seven assessed compliance with reminders to undertake appropriate laboratory testing,24,29,57,87,90,91,94 and one examined compliance with reminders about redundant tests. 92 The focus in three of the studies, the two CRCTs by Overhage and colleagues2,48 and the randomised crossover trial by McDonald and colleagues29 was on both reminders to order medication, screening, and other health-care procedures. In these three studies therefore only limited results were presented for laboratory test ordering alone.

The one study conducted in France, was undertaken in two accident and emergency departments, and assessed the impact of reminders to undertake the most appropriate radiological tests across all the physicians involved in the study. 93 This study therefore assessed the impact of reminders on the number of tests that complied with guideline recommendations. 93 A summary of study characteristics from the eight studies that assessed the impact of reminders is displayed in Table 13.

The two CRCTs conducted by Overhage and colleagues24,48 were both conducted on six inpatient medical wards (three intervention and three control) at the Wishard Memorial Hospital, Indianapolis, IN, USA. Both of the trials therefore used the site-specific home-grown Regenstrief Medical Record System which interfaced with two CDSS that appear to have been developed specifically for the trials. 24,48 The first trial published in 1996 included 78 physicians treating 1622 patients (821 intervention group and 801 control group) on one of the six wards.

During the 6-month trial period, wards were randomised to either receive or not receive preventative care measure recommendations provided by the CDSS. Any patient who had at least one preventative care recommendation was eligible for inclusion in the intervention group. The preventative care recommendation reminders were based on 22 preventative care measures from the US Preventive Services Task Force recommendations, and included vaccination measures, prescription of prophylactic medication, as well as laboratory screening test reminders. 113 Only 10 of the 22 preventive care recommendations were related to the use of laboratory testing and therefore of relevance to the present review. Data on these 10 were therefore extracted and presented, and included: (1) cervical cytology, (2) mammography, (3) thyrotropin screen, (4) hepatitis B screen, (5) screening urinalysis, (6) cholesterol test, (7) human immunodeficiency virus (HIV) screen, (8) 24-hour urine protein test, (9) sickle cell screen, and (10) sexually transmitted disease (STD) screen.

While the trial groups were well balanced in terms of age, gender, ethnic origin, and primary discharge diagnosis, data were presented only for the overall trial groups, not just patients who received laboratory test preventative care recommendations. It should therefore be noted that there may be potential baseline imbalances within the intervention and control groups, which could bias the results for these specific outcomes from this trial.

In the second trial, published in 1997, 86 physicians were randomised (45 intervention group and 41 control group) to receive or not receive reminders to order suggested corollary tests or medications needed to detect or ameliorate adverse effects to any one of the selected 87 trial tests or treatments. Standard reference texts114 and drug package inserts were used to identify 87 trial target orders (76 drugs and 11 tests) that were paired with one or more corollary orders. For example, aminoglycoside prescribing was paired with peak and trough aminoglycoside levels, and warfarin with prothrombin time. The CDSS then issued a reminder during the 6-month trial period to intervention physicians when any of the 87 target orders were placed, to consider ordering the linked corollary tests or medications. When suggesting corollary orders, the CDSS took into account the status of the order (a new order or a revision of an old order); the time elapsed since the last time the order being suggested was written; and whether any orders for a new equivalent item had already been written.

Physicians were free to accept or reject the suggested corollary orders. All inpatients who had at least one order written that triggered a suggested corollary order were eligible for trial inclusion, with 814 patients included in the intervention group and 872 in the control group. Again, groups were well balanced at baseline between the two groups in terms of age, gender, ethnic group, and primary diagnostic group. However, as the trial considered compliance with reminders to order both corollary medications and laboratory tests, again only the data for compliance with the ordering of laboratory tests was extracted and is presented. Therefore, as patient sociodemographic data were presented for the overall trial groups (not by type of order) then it is again possible that there were baseline differences between the intervention and control groups that may potentially confound the results for the laboratory outcomes of interest.

The CRCT by Palen and colleagues90 like the trial by Overhage and colleagues24 also focused upon the impact of reminders when ordering medications on compliance with guidelines for laboratory test monitoring. The trial, conducted at 16 primary care sites within the group model managed care organisation Kaiser Permanente, included 207 physicians (104 intervention group and 103 control group) who ordered 34,242 prescriptions for study medication for 26,586 different patients during the 12-month trial period. Physicians were randomised to receive or not receive non-intrusive drug–laboratory reminders for 25 selected medications within the OCS. This information was specific for the individual medication and presented guidelines for appropriate baseline monitoring. The 25 medications chosen were selected based on the presence of US Food and Drug Administration (FDA) black box warnings, published clinical guidelines, and the potential for adverse clinical consequences related to lack of monitoring.

The guideline recommendations for laboratory test monitoring for each medication were developed from information presented in national and internal clinical guidelines, and through discussion with physicians, pharmacists, and clinician leaders within the health-plan group. These guidelines then formed the basis for the reminders that were presented via the CDSS, which was implemented within the existing proprietary system (Clinical Information System) that was developed in collaboration with IBM (Boulder, CO).

For each of the 25 target medications, prescribing data from the 12-month trial period were analysed to assess whether appropriate laboratory tests had been completed within either 2 weeks after the medication order or had been ‘recently performed’. ‘Recently performed’ tests were defined as those completed within 180 days before medication dispensing or 2 weeks after dispensing. Physicians were defined as having followed the laboratory monitoring guideline if results of completed laboratory tests were available for review within these time frames (i.e. 180 days pre-dispensing and 2 weeks post-dispensing).

Matheny and colleagues91 in their CRCT also assessed the impact of reminders on appropriate laboratory monitoring of maintenance therapy with a focus on the monitoring of potassium, creatinine, liver function, thyroid function, and therapeutic drug levels for appropriate medications. The specific medications assessed were selected for inclusion in the reminder system based on (1) prevalence of their use, and (2) potential morbidity associated with failure to perform appropriate laboratory monitoring. These were based on evidence-based guidelines that were reviewed for routine medication monitoring. 115–118 The specific 15 target drugs included in the system were:

• non-steroidal anti-inflammatory drugs (NSAIDs)

• angiotensin receptor blockers (ARBs)

• metformin

• potassium supplements

• potassium sparing diuretic

• thiazide diurectic

• angiotensin-converting enzyme inhibitors

• HMG (3-hydroxy-3-methyl-glutaryl)-CoA reductase inhibitors (statins)

• thyroxine

• carbamazapine

• ciclosporin

• phenobarbital

• phenytoin

• proc-NAPA

• valproate.

The interval chosen for appropriate monitoring was annual, and therefore the trial included all outpatients (total n = 1922; 924 intervention group and 998 control group) seen during the 6-month study period on one or more of the 15 target study drugs for at least 365 days for which no relevant laboratory monitoring tests were conducted in the preceding year. The two groups were reasonably well matched in terms of age, gender, race and insurance status, and these factors were included in the analytic model to assess appropriate laboratory testing rates. Physicians from the Partners Healthcare system, which included two academic teaching hospitals (Brigham and Women’s Hospital and Massachusetts General Hospital), and a number of community hospitals and outpatient clinics were randomised from a total of 20 sites, either to receive reminders (n = 145 physicians) or for reminders to be suppressed (n = 158). The two physician trial groups were well balanced at baseline in terms of both age and gender.

The RCT conducted by Bates and colleagues,92 ‘a randomised trial of a computer-based intervention to reduce utilization of redundant laboratory tests’, was undertaken at Brigham and Women’s Hospital, Boston, MA, USA and included 5,700 inpatients in the intervention group and 5,886 in the control group admitted during the study period (between 28 June and 30 October 1994). Thirteen specific tests were chosen as appropriate candidates for redundant reminders either because they were commonly ordered or because the marginal cost of performing the test was high. The specific candidate tests were chemistry-20 profiles; urinalyses; urine, sputum and stools cultures; serum digoxin, tobramycin, aminophylline, vancomycin, and gentamicin levels; Clostridium difficile cultures;119 and fibrin split products.

Test-specific intervals within which a second test was considered redundant, were based on a review of the literature, and evaluated retrospectively by applying them to a random sample of patients. 120 ‘Target tests’ were defined as tests repeated earlier than the test-specific interval, which for all tests except fibrin split products were 20 hours. The test specific interval for fibrin split products was 8 hours. For digoxin and aminophylline levels, reminders were sent only if the first test result was within the reference range. A patient admission formed the unit of randomisation with physicians entering orders for both intervention and control patients using the home-grown site-specific Brigham and Women’s Hospital OCS.

In the intervention group, if a test had previously been ordered within its test-specific interval, the physician received a reminder that the test had been performed recently or was pending, and the result given if available. For the control group, redundancy was determined in the same way, but the reminder was suppressed. In the intervention group, when a reminder was delivered, the default response was to cancel the test. Physicians could override a reminder, but were required to choose from a menu of reasons for the override. For technical reasons, checking for redundancy was performed only for tests ordered from the main OCS screens and not for tests ordered using order sets or templates. Consequently, of the redundant laboratory tests performed, only 44% had an associated computer order. In total the trial included 5059 admissions (2478 in the intervention group and 2581 in the control group) who had at least one of the specified target tests within the 15-week trial period, with 13,425 study tests ordered in the intervention group versus 13,847 in the control group. Both groups were well matched at baseline in terms of age and gender. No details on sociodemographic data for the physicians were reported, or the number involved in the trial.

The randomised crossover trial by McDonald and colleagues,29 like the two CRCTs conducted by Overhage and colleagues,24,48 was undertaken at the Wishard Memorial Hospital, Indianapolis, IN, USA. However, the trial was undertaken in an outpatient population, and the CDSS system used, although part of the site-specific hospital system, is unlikely to be comparable to that assessed in the other trials due to considerable differences in the years in which the studies were conducted. It should also be noted that as the trial was conducted in 1980 the CDSS is likely now to be obsolete, and not of direct relevance to service providers needing to implement CDSS within the NHS at the present time.

The aim of this trial was to assess whether the CDSS that was designed to detect and remind physicians about clinical events that might need corrective action significantly increased response rates to events. The trial, which included 31 care providers (interns, residents and practice nurses) lasted 15 weeks: 5 weeks control period (no reminders sent); 5 weeks study period 1 (S1) and 5 weeks study period 2 (S2). During S1 care providers were sent reminders about the detected conditions alone, and during S2 were sent reminders plus bibliographic citations supporting the reminders. The correct action to the reminder was predetermined and compliance assessed. The three study periods, were presented in six possible temporal sequences, with care providers randomly assigned to a sequence. No information on patient sociodemographic status or presenting conditions was reported.

O’Connor and colleagues57 undertook a 5-year longitudinal study (CCT) to assess whether implementation of an electronic medical record with CDSS or the medical record system alone improved the process or outcomes of care in patients with an established diagnosis of diabetes mellitus. The trial was conducted in two clinics at the HealthPartners Medical Group, MN, USA and included all patients with a diagnosis of diabetes at study baseline (1996) in both clinics. The clinic that the patient attended was then identified in 1998 and 2000 based on administrative data. Patients were only eligible for inclusion in the analyses if they attended their original study clinic in all 3 years, with a total of 57 patients included in the intervention group and 65 in the control group. Patient baseline characteristics were well balanced between the two clinics in terms of age, gender, and Charlson comorbidity score. 121 As modified by Deyo and colleagues,122 and Rush and colleagues. 123 No sociodemographic data on the physicians participating in the trial were presented, but physicians in both clinics participated in the same diabetes-related care improvement activities within the medical group over the trial period. Similarly, both clinics had access to diabetes specific registries, in-clinic diabetes teaching nurses for patient education, and a common diabetes clinical guideline developed by the Institute for Clinical Systems Improvement (www.icsi.org/).

The CDSS used was a commercial system developed by Epic Systems (Madison, WI) that provided reminders to physicians when a patient had no HbA_1C test within 6 months, no urine microalbuminiuria test within 1 year, had blood pressures of ≥ 130/85 mmHg, low-density lipid (LDL) levels of ≥130 mg/dL, HbA_1C levels of ≥ 8%, or no aspirin use if aged 40 years or older. The reminders were displayed on screen, but a response to them was not obligatory. In the control group the reminders were suppressed. The electronic medical record system used a Windows interface and a Visual Basic (Microsoft Corp, Redmond, Washington, DC, USA) operating system linked to laboratory test results and pharmacy databases.

The process of care measures included the number of HbA_1C tests and LDL cholesterol tests conducted for patients in each clinic in 1996, 1998, and 2000. Additional process measures assessed whether patients met minimum thresholds for HbA_1C and LDL testing. Specifically, three threshold measures assessed whether the patient had at least two HbA_1C tests per annum; at least one LDL test per year; or at least two HbA_1C and one LDL test. Intermediate outcome measures included glycemic and lipid control, as assessed by HbA_1C and LDL test values in each of the three study years.

The only study conducted outside the USA was undertaken by Carton and colleagues,93 and aimed to assess the impact of guidelines on medial imaging referral practice in two hospital accident and emergency departments, the Hospital Ambroise Pare, Boulonge-Billancourt and Hospital de Pontchaillous, Rennes, France. The study was an ITS with an AB-AB-AB design, with the intervention implemented with three control periods and three intervention periods running alternatively over 6 months, with no delay between periods, and each period 1 month in length. The study included all physicians working in either emergency department who ordered a radiological examination within the study period. The number of physicians included in the study and details of their sociodemographic status were not reported. During intervention periods the CDSS displayed the appropriate guideline recommendation on screen for the patient’s indication; if the request did not conform to the guidelines, confirmation was requested before submitting the request. The request could still be fulfilled even if it was not in agreement with the guidelines. During the control periods, radiological requests were recorded, but no reminder displayed.

The CDSS knowledge base was written by the Collège des Enseignants de Radiologie de France (French Society of Radiologists) based on the results of a review of the literature, existing guidelines and the expertise of all the societies of radiologists and clinicians. During the study period a total of 15,086 patients were seen in the two departments, with 6,434 radiological requests recorded. Of these, 743 (11%) were discarded from the analysis because the radiological examination and/or clinical context had been profoundly changed by hand. The primary outcome measure was the number of requests complying with the guideline reminders, and secondary outcomes being the type of requests not in compliance.

The primary focus of the pre–post study by Steele and colleagues94 was on medication ordering, and drug–laboratory interactions. The primary outcome measure was therefore the number of medication orders cancelled after a reminder for a drug–laboratory interaction was displayed. The secondary outcome measures were the number of times the indicated laboratory test(s) were ordered once a reminder had been displayed, the number of associated test(s) ordered when an ‘abnormal labs’ message was displayed, and the number of appropriate test(s) ordered when a ‘no labs’ reminder was given. The study was conducted at the Sam Sandos Family Health Clinic, Denver Health outpatient primary care clinics, Denver, CO, USA. The study consisted of a 17-week pre-intervention period, and a 21-week intervention period, with a total of 19,076 patients seen during this 9-month period. This provided a total of 54,206 patient visits with medications ordered on 17,444 (32%) of visits. The rule processed on 49% of all medication orders during the entire study period. During the post-intervention period, a reminder was displayed to the care provider for 11.8% (1,093 out of 9,274) of the times the rule processed. Among these reminders, 5.6% were for only ‘missing laboratory values’, 6.0% were for only ‘abnormal laboratory values’, and 0.2% were for both types of reminders. This means that there were 14,297 medication orders across the study period where the rule was triggered, but it did not meet the criteria to display a reminder. All provider staff were allowed to enter medication orders, including physicians, allied health-care providers (nurse practitioners, physician assistants), and residents. No further information was reported on health-care provider demographics. All registered patients were eligible for inclusion in the study.

The CDSS was developed in collaboration with Thomson Micromedex and Siemens Medical Solutions, and used commercially available rules developed in Ardex Syntax language as Medical Logic Modules, which were then modified to meet local needs. The study focused upon rules that were determined as being the most appropriate to addressing patient safety in an outpatient setting. The rules covered the following five areas:

1. potential drug-induced hyperkalemia (covering interactions with captopril, lisinopril, spironolactone, and enalapril)

2. hypokalemia (covering interactions with chlorthalidone, furosemide, hydrochlorothiazide, metolazone and ethacrynic acid)

3. thrombocytopenia (covering interactions with ranitidine)

4. elevated creatinine levels (covering interactions with amiloride, chlorthalidone, enalapril, furosemide, hydrochlorothiazide, lisinopril, metolazone and spironolactone)

5. elevated transaminase (covering interactions with atorvastatin, citalopram, fluoxetine, paroxetine, rosiglitazone and sertaline).

For each drug–laboratory interaction, rules were written identifying medications, routes of administration, and abnormal laboratory threshold levels for inclusion in the rule. The laboratory cut-off values for triggering a reminder were the same as for the Denver Health abnormal laboratory reference ranges. In addition, a determination was made for each medication as to whether a reminder should be provided for an abnormal laboratory value only, an abnormal or missing laboratory value, or whether despite an association with a laboratory abnormality, no reminder would be displayed. In response to reminders, CDSS users could decide to order, revise or delete the medication order. They could also order any rule-associated laboratory tests. Users did not need to respond to the reminder, but needed to select ‘Continue’ to proceed with the drug ordering session.

In the 17-week pre-intervention period, the rules were turned on in the background, but no reminders were displayed to users. Baseline ordering behaviour was then compared in a pre-post design with ordering behaviour during the 21-week intervention period.

The UPP study conducted by Abboud and colleagues87 was undertaken in paediatric inpatients at the Cincinnati Children’s Hospital Medical Centre, Cincinnati, OH, USA. The study consisted of a 3-month pre-intervention period followed by a 3-month intervention period, with a total study duration of 6 months. The study was specifically aimed at assessing the impact of a CDSS corollary order for aminoglycoside laboratory blood monitoring levels in all patients who received aminoglycosides for 4 or more days duration during the study period, as this was identified as being suboptimal at study baseline. This interval was chosen by investigator consensus as the minimum duration of therapy that would require monitoring of aminoglycoside levels. The CDSS was developed and implemented in the existing hospital information system (invision®, Siemens Medical Solutions, Malven, PA, USA).

During the intervention phase, immediately after placement of an order for aminoglycosides, the physician was reminded to check peak, trough, peak and trough, or random blood levels for the drug prescribed. This could be undertaken during the same session as the prescribing session. During the pre-intervention phase no reminders were presented on screen.

The study included 159 courses of antibiotic therapy of 4 or more days duration (n = 125 patients) in the pre-intervention phase, and 177 (n = 150 patients) in the intervention phase. No specific patient demographics were reported, but groups were well matched in the pre- and post-intervention phases in terms of the number of courses of aminoglycosides prescribed, total number of laboratory results obtained, and the number of laboratory results per patient. For the analyses, all antibiotic levels ordered were utilised in assessing the response to the reminder, but only true peak and trough levels were used to assess toxicity and subtherapeutic values. These were defined according the hospital laboratory guidelines, with peak and trough levels further divided into toxic or subtherapeutic. In the study a peak level was predefined as a level obtained 50–120 minutes after drug administration, and a trough defined as being obtained 0–120 minutes prior to drug administration.

Outcomes

Study quality

CDSS characteristics

A summary of the key characteristics of the CDSS used in the 10 studies are displayed in Table 14. The specific CDSS used in the studies were only reported in four of the studies,57,87,90,94 but in the two CRCTs by Overhage and colleagues24,48 and the study by McDonald and colleagues29 would appear to be the home-grown/inhouse OCS and CDSS developed by Wishard Memorial Hospital, Indianapolis, IN, USA. Likewise, in the CRCT by Matheny and colleagues91 and the RCT by Bates and colleagues92 the system would again appear to one of the site specific ones, this time belonging to Brigham and Women’s Hospital, Boston, MA, USA. In the four studies that reported the systems used, the CRCT undertaken by Palen and colleagues90 used a study-specific CDSS (Clinical Information System) developed in collaboration with IBM (Boulder, CO, USA) that was implemented within the existing Kaiser Permanente proprietary system; the CCT by O’Connor and colleagues57 used what appears to be a commercially available diabetes mellitus-specific CDSS developed by Epic Systems (Madison, WI, USA); and the two pre–post studies by Steele and colleagues94 and Abboud and colleagues87 used study-specific CDSS that were developed in collaboration with Thomson Micromedex and Siemens Medical Solutions using commercially available Medical Logical Modules modified to meet the needs of the Denver Health laboratory,94 and a CDSS developed by invision®, Siemens Medical Solutions, Malven, PA, USA that was then implemented within the existing hospital information system. 87 The CDSS used in the ITS by Carton and colleagues93 was not reported.

In the four studies that reported the CDSS reasoning methods, these were all based on discrimination rules. 24,48,91,94 Where reported the CDSS knowledge base was based upon either reviews of the literature,92,93 guidelines,48 standard reference books, and drug packet inserts,24 or existing database information. 94 The CDSS output in all studies was the display of reminders.

In relation to the 14 features of CDSS proposed by Kawamoto and colleagues16 the CDSS was not reported as being piloted with users prior to implementation in any of the 10 studies, but user instructional training at the time of implementation was reported in four. 57,90,92,100 The CDSS in all 10 studies was integrated as part of the OCS, and provided output automatically as part of the consultation workflow. Only in the UPP study by Carton and colleagues was there a need for additional data entry by the physician,93 and only in this study and the CRCT by Overhage and colleagues48 was a reason requested for not following the CDSS reminders. None of the studies required the user to note agreement with the reminder before implementation. The study by O’Connor and colleagues57 was the only one in which a recommendation rather than just a reminder was issued. In none of the studies was the reminder justified by recourse to the CDSS reasoning methods or evidence upon which these were based. Additionally, it would appear that local system users were only involved in the CDSS development process in the study by Steele and colleagues. 94 The data provided by the CDSS was used by the physician alone, and in all studies did not appear to provide periodic summaries of performance feedback. Two studies, those by O’Connor and colleagues57 and Abboud and colleagues87 both combined CDSS implementation with concomitant coeducation of users. None of the other studies deployed any concomitant cointerventions.

Results

Summary of studies assessing the impact of the display of reminders

Quantity and quality of the studies

Assessment of the display of restricted lists was examined in one ITS study by Rosenbloom and colleagues95 and one CPP study by Poley and colleagues. 96 The study by Poley and colleagues96 also included a cost-comparison analyses, the results of which are reported in Chapter 6, Systematic review of economic evaluations. Both studies targeted specific tests, with the ordering of serum magnesium level tests being the subject of the ITS by Rosenbloom and colleagues95 and blood test ordering targeted by Poley and colleagues. 96 The area of impact in the test ordering process targeted in both studies was therefore test volumes. 95,96 The ITS by Rosenbloom and colleagues95 was conducted on 30 of the 33 inpatient wards at the Vanderbilt University Hospital, Nashville, TN, USA over a 6-year period (1 January 1998 to 31 December 2003). The specific CDSS evaluated was the WizOrder Care Provider Order Entry System, a home-grown system developed by the medical centre. The system assessed in the study was a non-commercialised form of the software code used at the hospital. All patients admitted to any ward where the CDSS was implemented over the study period were eligible for inclusion; with a total of 194,192 patients admitted. The CDSS users were all physicians, nurse practitioners or medical students working within the specific wards. No further data on CDSS user sociodemographic variables were presented.

The study consisted of three different CDSS intervention protocols that were implemented sequentially over the study period. The Vanderbilt University Medical Centre Resource Utilization Committee firstly identified two ‘normal’ ranges for serum magnesium test results that were considered appropriate. One was based on the institutional laboratory’s statistically normal range, 1.5–2.5 mg/dL, and other represented a ‘physiologically appropriate’ range (i.e. serum magnesium was unlikely to directly cause clinical sequelae if within this range) of 1.0–3.9 mg/dL. These were then implemented within the CDSS, with changes to the three different interventions made based on the results of the previous intervention.

At study baseline (1 January 1998 to 4 December 1999), no protocols were in place restricting the frequency, context or volume of serum magnesium test ordering. The first CDSS intervention (implemented from 5 December 1999 to 21 March 2000), targeted all open-ended laboratory and radiology orders, with tests scheduled more than 72 hours into the future flagged, and users prompted to consider discontinuing the order. The second intervention, which was implemented from 20 June 2000 to 30 November 2001, was also a broad-based CDSS intervention, that globally addressed the ordering of multiple laboratory tests simultaneously. The specific tests addressed were magnesium, calcium, and phosphorus.

The intervention included a graphical display of patients’ recent serum magnesium, calcium, and phosphorus test results, educational material outlining indications for magnesium testing, and test interpretation. This intervention also limited orders to one test per order (i.e. no recurrent testing was possible). CDSS users could bypass the intervention only by ordering magnesium testing from disease-specific order sets or by specifically ordering a single magnesium test (rather than recurrent testing) from the standard OCS. The third intervention, implemented from 1 December 2001 to 31 December 2003, focused only on the ordering of magnesium tests. This intervention included a graphical display of patients’ most recent serum magnesium result, and a graphical display of a calculated corrected magnesium value. Test volumes were also limited to one test per order. In addition users had to enter a reason for testing after reviewing a list of indications. CDSS users could bypass the intervention by ordering magnesium tests from a disease-specific order set. Both of the second and third interventions implemented therefore included a concomitant CDSS intervention, such as the display of educational material, and were not limited to just the use of test restrictions. Follow-up within the study was conducted at 11 months, 27 months; 47 months and 72 months. 95

The CPP study by Poley and colleagues96 was conducted in 134 primary care practices in the Netherlands, with the study consisting of a 6-month pre-intervention period and a 6-month post-intervention period. The total length of study follow-up was therefore 12 months. The study included 234 primary care physicians (159 in the intervention group and 75 in the control group), with study inclusion criteria including submission of greater than 80% of blood test orders to one of the 27 laboratories participating in the study, and use of one of three information systems (Elias, MicroHIS, or Promedico) for which the CDSS was developed. There were no statistically significant differences in baseline sociodemographic variables either between physicians in the intervention and control groups, or between physicians in the intervention group and national figures on physicians from the Netherlands Institute for Health Services Research. The CDSS was comprised of an optimal but restricted list of blood tests based on recommendations for blood test ordering for the patient’s indication, selected by the physician based on guidelines from the Dutch College of General Practitioners (http://nhg.artsennet.nl/). The GP could adhere to the proposed list or add or remove tests from the list as appropriate. Additionally, the GPs were not obliged to use the CDSS. The specific CDSS developed was a study specific system, that is available (in Dutch only) from the authors, and is integrated with the three OCS specified above as in the study. 96 A summary of the study characteristics from the two studies is displayed in Table 21.

Outcomes

Study quality

In both studies, study eligibility criteria were adequately reported,95,96 and groups were balanced in terms of sociodemographic variables at baseline in the pre–post study by Poley and colleagues. 96 In both studies the intervention appeared to be implemented independently of other changes in the study period, which is of particular import in the ITS study conducted by Rosenbloom and colleagues,95 and in the study by Poley and colleagues was controlled for by use of a separate group that were not exposed to the CDSS intervention. 96 In both studies therefore, steps were taken to limit bias from exposure to other concomitant interventions or independent secular changes over time.

The intervention could not influence the methods of data collection in either of the studies, and the primary outcome measure of the number of test orders was reliable. 95,96 In the ITS study by Rosenbloom and colleagues95 sufficient data points over time were reported for reliable statistical inference, and formal tests for trends were conducted. 95 In both studies statistical analyses were appropriate, and in the controlled pre–post study by Poley and colleagues comparisons of pre- and post-intervention periods, and between group comparisons were conducted. 96 However, data were not analysed on an ‘intention to provide or communicate information’ basis in this study, and the rate of attrition in the intervention group (23%) was reasonably high. This has the potential to bias the results, and no sensitivity analyses were conducted to explore differences between study completers, and those who dropped out. 96 In the ITS study by Rosenbloom and colleagues95 analyses were conducted for all instances of test ordering, and results are therefore unlikely to be biased.

CDSS characteristics

A summary of the CDSS characteristics in both studies is displayed in Table 22. In the ITS by Rosenbloom and colleagues95 the specific CDSS evaluated was the WizOrder Care Provider Order Entry System, a home-grown/inhouse system from Vanderbilt University Medical Centre. In this particular study a non-commercialised form of the software code was implemented. The CDSS used in the controlled pre–post study by Poley and colleagues was not reported, but a Dutch language version of the CDSS is available on request from the authors. Neither of the studies reported the CDSS reasoning methods, but the information used in the CDSS appeared to the normal reference ranges for serum magnesium test results in the study by Rosenbloom and colleagues95 and information from the relevant guideline in the study by Poley and colleagues. 96

In both studies the output format was primarily restricted lists, but also included restrictions on forward ordering, a graphical display of previous test results, and education material as concomitant interventions at different phases of the ITS by Rosenbloom. 95 Neither of the studies reported the time to complete the CDSS, or whether pilot testing and user training were provided prior to implementation. Both the CDSS were integrated with OCS and provided output at the time and location of decision making. Neither of the systems required the additional input of information. Additionally as output was limited to the display of restricted lists only and no specific recommendations were provided, the user was not required to document a reason for not following the CDSS recommendations. It was unclear in both studies whether the display of restricted lists was likely to promote action rather than inaction on the part of the physician. The data provided by the CDSS was used by the physician alone, and did not appear to provide periodic summaries of feedback performance.

Results

Summary of studies assessing the impact of the display of restricted lists

Two studies, one ITS study conducted on inpatients in the USA, and one CPP study conducted in general practice in the Netherlands assessed the impact of the display of restricted lists. The aim in both of the studies was to limit unnecessary test orders, and therefore the outcomes of interest were test volumes. Both of the studies focused on specific test types with Rosenblooom and colleagues95 focusing primarily on serum magnesium test orders, and as secondary outcomes calcium and phosphorus test instances. Poley and colleagues96 targeted a range of blood tests.

Results from the ITS by Rosenbloom and colleagues,95 generally showed a significant reduction from baseline in mean weekly requests for serum test orders, from 539 at baseline, to 380 after the implementation of the first intervention, and 277 after the third. Paradoxically, there was a significant increase in test order rates after implementation of the second intervention to 491. Net requests of magnesium test orders per patient also followed this trend. Concurrent calcium and phosphorus test ordering with serum magnesium tests also showed a similar pattern, both decreasing significantly from baseline after implementation of the first intervention, increasing significantly after introduction of the second, and then decreasing significantly after the third. Overall, therefore, it would appear in this study that while the CDSS had the potential to regulate and limit unnecessary serum magnesium test ordering, other concomitant CDSS interventions may interact with this aim, and have the potential to paradoxically increase test ordering rates.

Limited results from the CPP study by Poley and colleagues96 indicated that implementation of an optimal but restricted list of blood tests within the CDSS did not significantly decrease the mean number of laboratory request forms over and above the use of OCS alone. However, there was a small (–0.38) but significant decrease noted in the number of test requests per form in favour of the intervention group. No results were reported for the overall number of blood tests conducted. A summary of the results of the two studies is displayed in Table 23.

Quantity and quality of the studies

The effect of the CDSS providing a recommendation was assessed in seven studies;32,50,97–100,104 two CRCTs published by Hobbs and colleagues32 and Cobos and colleagues50 respectively; one RCT conducted by Apkon and colleagues;97 and four pre–post studies undertaken by Bassa and colleagues,98 Sanders and colleagues,104 Nightingale and colleagues99 and Boon-Falleur and colleagues. 100

Across the studies, two each were conducted in the UK,32,99 Spain,50,98 and the USA,97,104 with the remaining study being undertaken in Belgium. 100 Three of the studies which all focused on the management of hypercholesterolaemia were undertaken in primary care settings,32,50,98 (one from the UK,32 and two from Spain)50,98 while of the remaining four studies conducted in a secondary care setting, one assessed the impact of CDSS recommendations on the number of health-care opportunities fulfilled in terms of the number of laboratory and radiology screening tests undertaken,97 one examined the impact on test ordering patterns of neuroradiology head imaging studies,104 while the further two studies assessed the impact on the number of laboratory test requests in patients either being assessed for or having undergone a liver transplantation. 99,100 The RCT which assessed the impact of recommendations on the number of laboratory and radiology screening test opportunities undertaken was conducted in the USA at two military treatment facilities97 and reported the number of health-care opportunities fulfilled for a wide range of health problems including vaccinations, pharmacological treatments, smoking cessation, and diet and exercise counselling. As only the number of opportunities fulfilled, in terms of laboratory test screening and back pain imaging were of relevance to the current review, only data on these outcomes were extracted and are reported. A summary of study characteristics from the seven studies that assessed the impact of recommendations is displayed in Table 24.

The first CRCT which assessed the impact of the provision of recommendations was undertaken in 25 primary care practices (21 intervention and 4 control) in Birmingham, UK, by Hobbs and colleagues. 32 The aim of the 9-month trial, which included a 3-month historical baseline control period and a 6-month intervention period, was to examine the effect of CDSS on the management of hyperlipidaemia in patients not previously diagnosed with the problem. The primary outcome of interest in relation to this review was lipid test rates between intervention and control groups, although changes in prescribing and referral practices were also reported. The CDSS used was the Primed system designed for use in general practice by Wolfson Research Laboratories, University of Birmingham, UK. The software was a rule-based system, which provided an initial screening prompt for capture of patient sociodemographic and cardiovascular risk factor data. This also included the input of the patients’ current cholesterol level. The patients’ coronary risk score was then displayed on screen, with a score greater than 10 being in the top quintile for risk of a coronary event within the next 5 years.

Based on the risk score, the CDSS provided advice on patient management, with the rules underpinning the recommendations derived from a protocol developed by a lipid specialist. Overall, the trial was subject to high rates of attrition with eight clusters (seven intervention and one control) withdrawing (the total number of physicians within the trial was not reported). Furthermore, only limited results were reported, and all comparisons were analysed as a change from the 3-month baseline period, rather than as between group comparisons, thus limiting their utility in assessing the effects of the CDSS in conjunction with OCS compared to OCS alone.

The second non-inferiority CRCT conducted by Cobos and colleagues50 in general practices in Spain (mainly drawn from the Catalonia region) aimed to assess the cost-effectiveness of CDSS for adapted recommendations based on guidelines from the European Society of Cardiology and other societies for Hypercholesterolemia Management. 124 The 12-month pragmatic trial, included 44 practices (22 in the intervention group and 22 in the usual care group) with a total of 2191 patients (1046 in the intervention group and 1145 in the usual care group) with a diagnosis of hypercholesterolemia. This was defined as a pre-treatment total cholesterol concentration > 200 mg/dL, and < 400 mg/dL. Patients who lacked a baseline lipid profile were excluded from all analyses. Patient groups were well matched at baseline in terms of age, gender, mean BMI, and other concomitant risk factors for hypercholesterolemia. No sociodemographic data on the physicians participating in the trial was reported.

The CDSS was based on adapted algorithms that were implemented in the intervention group and withheld in the usual care group that issued recommendations on therapy, frequency of follow-up visits, and laboratory tests according to the patient’s cardiovascular risk factors LDL-C goals. Physicians were free to either adopt or ignore the CDSS recommendations. Adherence to the guideline was monitored by the CDSS and a reason requested in case of any discrepancy between the treatment recommended and prescribed. A concomitant intervention, of the provision of items such as table cloths and fridge magnets with relevant promotional messages was also undertaken in intervention group practices, but withheld in the usual care practices. Only data on the number of laboratory test analyses were extracted and are therefore reported in this review.

Like the trial conducted by Hobbs and colleagues32 this study was subject to a high rate of missing post-baseline data with 26% of the intervention group and 28% of the control group having no post-baseline lipid profile. Additionally, only 59% of the intervention group and 47% of the usual care group had a follow-up of 9 months or more. However, appropriate sensitivity analyses to missing data (no post-baseline assessment and < 9 months follow-up) were conducted. Among patients lost to follow-up after the first (baseline) visit no change in lipids or coronary vascular risk (CVR) was assumed. This assumption was then tested in two different analyses, one including patients with at least one post-baseline assessment (per protocol analysis) and patients with a length of follow-up of at least 9 months.

The RCT conducted at the Ireland Army Community Hospital and Clinic, Fort Knox, KT, USA, and the Mayport Branch Health Clinic, Mayport, FL, USA, within two military treatment facilities by Apkon and colleagues97 aimed to assess the impact of the use of CDSS on quality of patient care. This was defined as the total percentage of any of 24 health-care quality process measures (opportunities to provide evidence-based care) that were fulfilled within 60 days of a patients’ index visit. The trial measured and reported a wide range of health-care opportunities, but as only the results of those that have an impact on laboratory testing or radiology imaging rates are relevant to the present review, only data for these outcomes were extracted and reported.

A total of seven outcomes were deemed relevant to this review:

1. cervical cancer screening

2. screening for Chlamydia

3. colorectal cancer screen

4. lipid level testing

5. back pain imaging

6. screening for diabetes using glycosylated haemoglobin levels

7. screening for lipid abnormalities.

In addition laboratory test resource and diagnostic test imaging consumption were reported.

These were calculated from the Centers for Medicare and Medicaid Services 2003 fee schedule using the relative value unit conversion rate of US$36.8. The 60-day trial included 4639 health-care opportunities (2374 in the intervention group and 2265 in the control group) among study patients at their index visit. Patient characteristics were reasonably well balanced at baseline in terms of age, gender, and type of visit (acute, established, routine, wellness or other). However, as data were reported by trial group rather than by patient indication, it is possible that there were baseline imbalances between the two groups which could potentially bias results. No sociodemographic data or the number of physicians involved in the trial were stated.

The CDSS intervention which used the DSIT Problem-Knowledge Couplers (PKC Corp, Burlington, VT, USA) involved the use of a number of ‘Couplers’ which were available for a wide range of preventive and acute/chronic disease management needs. Patients entered data into the Coupler appropriate for their complaint, or when no condition-specific Coupler was appropriate, a generic medical history and screening Coupler, prior to seeing the physician. This took approximately 20 minutes. Physicians treating intervention group patients could enter additional information before reviewing Coupler outputs outlining investigation or treatment options. Patients in the control group had no exposure to Couplers. In total there were 667 health-care opportunities (325 in the intervention group and 342 in the control group) from the seven of relevance to this review.

Of the four pre–post studies, the first conducted by Bassa and colleagues98 was undertaken at the Vila Olimpica Primary Health Care Centre, Barcelona, Spain and included 500 patients randomly selected from the Primary Health Care centre database with hypercholesterolemia. These patients had a median age of 67 years, 65.8% (329/500) were female, 18% had a concomitant diagnosis of diabetes mellitus (90/500), 16.6% (83/500) were current smokers, and 52.4% (262/500) had a sedentary lifestyle. In terms of the number of cardiovascular risk factors per patient at baseline in the study, 4.4% (22/500) had none, 34.8% (174/500) had one, 53.4% (267/500) had two, and 7.4% (37/500) had more than two. The number of physicians involved in the study or sociodemographic data on them were not reported.

The aim of the study was to assess the impact in terms of effectiveness and costs of CDSS and to implement a practice guideline for patient management. The study was carried out over a 2-year period, with 1-year pre-implementation of the CDSS and 1-year post-implementation. The CDSS recommendations were based on the Sociedad Española de Medicina de Familia y Comunitania’s clinical guidelines for dyslipidemia management and cost-effectiveness data published in a meta-analysis. 125,126 Based on the patient’s data (personal history of cardiovascular disease, cardiovascular risk factors, and lipid profile), the CDSS established the therapeutic objectives in terms of LDL cholesterol levels and issued testing, therapeutic and follow-up recommendations for the patient. Physicians were free to accept or decline the recommendations, but were prompted for a reason when they declined. The recommendations included dietary treatment, lipid-lowering drugs, monitoring of hepatic and muscular enzymes and a recommended date for the follow-up visit. As only data on the number of lipid-profile tests conducted and their costs are of relevance to the present review only these data were abstracted and are reported. Data associated with the mean cost of undertaking laboratory assessment tests and the data sources and method used to calculate these are reported in Chapter 6, Systematic review of economic evaluations.

The second pre–post study undertaken by Sanders and colleagues104 was conducted at the Vanderbilt University Medical Centre, Nashville, TN, USA, using a module specifically designed for use in the University-specific WizOrder entry system. The study consisted of a 9-week control (pre-intervention) period and an 8-week intervention phase. The aim of the study was to assess the effects of guidelines implemented within the CDSS for the ordering of neurological head imaging studies on test ordering patterns and guideline compliance.

All physicians, nurses, medical students and receptionists who entered an order via the system for one or more head CT or brain MRI scans during the study period were eligible for inclusion in the study, with a total of 1446 orders included in the analyses (742 pre-implementation and 704 post-implementation). To develop the CDSS a list of common indications for ordering an imaging examination of the head or brain was created based on prior free text indications at the time of order entry, historical International Classification of Diseases, Ninth Edition (ICD-9) coding data and local clinical guidelines for the tests. These were then mapped to ICD-9 codes, and for each indication, the most appropriate imaging test determined.

The CDSS required input of the patients’ acuity and indication by the user and provided a recommended test (head CT without contrast; head CT with contrast; head CT with and without contrast; brain MRI without contrast; and brain MRI with and without contrast). If a suggestion was given, this choice was defaulted. The user was able to override the recommendation and select any of the listed studies, but had to type the reason for doing so. If no indication for requesting the test was given by the user or ‘other’ was chosen, no CDSS recommendation was provided. Analyses were then conducted between study periods to evaluate changes in the distribution of test ordering patterns.

The last two pre–post studies were both conducted in secondary care settings involving patients either being assessed for, or undergoing, liver transplantation. 99,100 The first of the two studies, by Nightingale and colleagues99 was undertaken in adult patients at the Supraregional Liver Unit, The Queen Elizabeth Hospital, Birmingham, UK. The study aimed to assess the effects of a CDSS protocol management system on the number of tests, costs, and the appropriateness of laboratory investigations requested. The study consisted of a 1-year pre-implementation phase during which 654 patients were assessed, and a 1-year post CDSS implementation phase during which 833 patients were assessed. Patient categories in terms of the numbers undergoing initial assessment, reassessment, transplant, post-transplantation or being an emergency were fairly well matched between the two study periods. No socio-demographic data on the physicians involved in the study or their numbers were reported. The study specific CDSS was developed by Wolfson Computer Laboratories at the hospital, and aimed to implement the unit’s existing investigation protocols developed by senior clinicians. These were based on information regarding the category of the patient’s clinical state (e.g. assessment, transplant) and the use of the latest test results to propose the laboratory investigations to be performed on the following day. The system was based upon a combination of static and dynamic rules. Static rules were those that applied to all patients with a certain classification for a certain number of days, and dynamic rules were those which used the results of previous laboratory results to determine which investigations to propose. Once the physician had viewed the proposed tests they were free to accept or modify them as required.

The same CDSS as used in the study by Nightingale and colleagues99 was adapted to a multilingual version and exported for use at the Paediatric Liver Transplantation Unit, Cliniques Universitaires Saint-Luc, Université catholique de Louvai, Brussels, Belgium by Boon-Falleur and colleagues. 100 This study, like that by Nightingale and colleagues,99 aimed to assess the impact of the system on the number of laboratory tests performed pre-implementation and post-implementation. However, this study only included paediatric inpatients who were undergoing either a pre-transplant assessment protocol (n = 183; 32 pre-intervention and 151 post-intervention) or transplant protocol (n = 34; 10 pre-implementation and 24 post-implementation). No sociodemographic data on the patients in either study period were reported, and so it is not possible to comment on whether there were systematic differences in the patient populations in the pre- and post-intervention periods that may potentially confound the results. Likewise, no socio-demographic data on the physicians or the number involved in the study were reported.

The pre-implementation phase consisted of a 6-month period, but the length of post-intervention assessment was not reported. Additionally, the results were reported according to the number of tests per patient, rather than the number of tests per patient per day, which potentially confounds length of stay with number of tests, and is not an appropriate outcome measure.

Outcomes

Study quality

CDSS characteristics

A summary of the key characteristics of the CDSS used in the seven studies is displayed in Table 25. The specific CDSS used was only reported in three of the studies,32,97,104 these included the DSIT tool Problem-Knowledge Couplers (PKC Corp, Burlington, VT, USA) implemented within the existing Military Health System’s electronic medical system (Composite Healthcare System),97 the Primed system developed specifically for use in patients with hyperlipidemia in a primary care setting by Wolfson Research Laboratories, University of Birmingham, UK,32 and the WizOrder entry system, the home-grown/in-house system from Vanderbilt University Medical Center, Nashville, TN, USA. 104 In the other studies, three of the other four CDSS appeared to have been developed and implemented specifically for the sites and specialities in which it was used (in two studies screening for hyperlipidemia50,98 and in the third for the assessment and management of patients undergoing liver transplantation). 99 This CDSS was developed again by Wolfson Research Laboratories, University of Birmingham, UK, and adapted to a multilingual version for use in the study by Boon-Falleur and colleagues,100 again for the management of liver transplantation patients.

In three of the seven studies the CDSS reasoning methods were discrimination rules,32,99,100 and in a further two clinical algorithms. 50,98 Reasoning methods were not clear in the remaining two studies. 97,104 The CDSS knowledge base was either expert opinion,32,99,100 clinical guidelines,98,104 or a combination of prior indications at the time of order entry, historical ICD-9 coding data and guidelines. 104 The information used in the CDSS was the patient’s medical history in one RCT which patients entered themselves into the appropriate Coupler,97 CV risk factors coupled with either cholesterol level or lipid profile in three studies,32,50,98 acuity and indication in the pre–post study assessing the impact of CDSS implemented guidelines for radiological imaging in one study,104 and not reported in the remaining two studies. 99,100 In these two studies it would, however, appear to have been based on the patients condition, history, and results of any previous laboratory tests. 99,100 The CDSS outputs in all the studies was the display of recommendations. Only the RCT by Apkon and colleagues97 reported the length of time to complete the CDSS Coupler which was 20 minutes and patient completed. In relation to the 14 features of CDSS proposed by Kawamoto and colleagues16 as being important to the success or failure of a CDSS, the CDSS was not reported as being piloted with users prior to implementation in any of the seven studies, and user instructional training was provided in only one. 32 In all seven studies the CDSS was integrated as part of the OCS, and provided output automatically as part of the consultation workflow. In three of the studies there was a need for additional data entry by the physician,32,99,100 while in a further study this was optional. 97 Additionally, in three studies there was a need for documentation for not following the CDSS recommendations,50,98,104 however, none of the studies required the physician to note agreement with the recommendations in order for these to be implemented. In none of the studies was a recommendation justified by recourse to the CDSS reasoning methods or evidence upon which these were based. Additionally it would appear that local users were directly involved in the CDSS development process in only two of the studies. 100,104 The data provided by the CDSS was used by the physician alone, and only in two studies was periodic performance feedback provided. 99,104 None of the studies deployed any concomitant educational cointerventions.

Results

In the CRCT by Hobbs and colleagues32 which assessed the impact of CDSS on screening patients with previously undiagnosed hyperlipedmia no point estimates for pre- and post-implementation rates of lipid test rates were reported. Likewise, no between group comparisons were conducted. The authors stated that in the 9-month post-implementation period the mean rate of testing was 4.4 tests/1000 population/month. This rate was not significantly different from the pre-intervention phase. However, the authors stated that there was a significant increase in the number of patients receiving a full lipid profile in the intervention phase, and a decrease in those having only a partial investigation compared with the baseline period (p < 0.05). Again the exact figures were not reported.

Similarly, in the 1-year CRCT conducted by Cobos and colleagues50 there were no significant differences between the treatment and usual care groups (n = 1046 and 1145 respectively) in the number of lipid assessments conducted per patient visit. This was 1.8 in the intervention group compared with 1.8 in the control group (p = 0.298). However, there was a significant increase in the number of patients receiving AST/ALT tests per visit in the intervention group, 1.4 compared to the control group, 1.3 (p = 0.033). No significant differences were observed in the number of CK tests performed per patient visit, with 0.54 and 0.24 conducted in the intervention and control groups respectively (p = 0.053).

The 60-day trial undertaken by Apkon and colleagues97 to assess the impact of CDSS Coupler recommendations on the number of health-care opportunities fulfilled 60-days after the patient’s initial index visit, showed no statistically significant differences between the treatment groups in terms of any of the seven opportunities related to laboratory or imaging tests screening. A break down of the number of opportunities fulfilled in each group by opportunity type is displayed in Table 26.

However, the authors reported a statistically significant increase in median laboratory test resource consumption in the intervention group of US$43 (range: 0–144) compared to the usual care group, US$31 (range: 0–139) (p = 0.04). No differences were observed between the two groups in terms of diagnostic test imaging consumption, with these being US$31 (range: 0–148) and US$29 (range: 0–127) in the intervention and control groups respectively (p = 0.26).

Bassa and colleagues98 from their 2-year pre–post study on the effects of the implementation of guidelines for the treatment of patients with hypercholesterolmia implemented via CDSS, reported no significant differences in the number of lipid tests carried out in the pre-and post-implementation phases of the study. At baseline a total of 773 tests were conducted per annum compared with 763 post-intervention (p = 0.59).

In the 17-week pre–post study by Sanders and colleagues104 examining the effects of implementation of a guideline for ordering head or brain imaging studies, there was a small significant decrease in the number of imaging tests ordered after implementation of the CDSS. This fell from 742 tests ordered in the pre-implementation phase to 704 tests post-intervention (p = 0.048). Compliance with ordering the CDSS recommended tests however, was still fairly low at only 60%.

The two pre–post studies by Nightingale and colleagues99 and Boon-Falleur and colleagues100 assessing the impact of CDSS recommendations for adult and paediatric inpatients pre- and post-liver transplantation protocols showed slightly disparate results. In the 2-year study by Nightingale and colleagues99 there was a significant (17%) overall reduction in the number of tests requested per patient day from 8.5 (SD 3.6) pre-intervention to 7.0 (SD 3.5) with the implementation of the protocols (p < 0.001). This reduction was consistent across all patient categories apart from patients undergoing routine reassessment in which an insignificant reduction from 4.8 (SD 2.7) to 3.7 (SD 2.7) tests per patient day was observed, and those undergoing an annual post-transplant review in which an insignificant 12% reduction in the number of tests ordered from 7.4 (SD 4.1) to 6.6 (SD 2.9) per patient day was observed. Additionally, there was an increase in the number of tests ordered for patients with emergency acute hepatic failure of 17%, from 6.7 (SD 3.8) at baseline to 7.8 (SD 4.0) per patient day post-implementation. A summary of the changes in the number of tests ordered per patient per day by patient category is displayed in Table 27.

The implementation of the CDSS protocols resulted in a significant 48% decrease in the number of out of hours tests requested per patient day, from a pre–post baseline of 0.31 to a post-intervention number of 0.16 (p < 0.001). Likewise, the median number of plasma urea and electrolyte tests (p < 0.05), bone profile tests (p < 0.001), and calcium tests (p < 0.005) were all significantly reduced. However, there was no significant reduction in the number of liver function tests conducted, and a minor non-significant increase in the number of others tests undertaken. The overall reduction in the number of tests conducted was reflected in direct laboratory costs per patient days with a significant 28% reduction observed (p < 0.001).

Results of the study by Boon-Falleur and colleagues100 in contrast to those of Nightingale and colleagues99 showed a 13% increase in the number of tests ordered per patient stay for patients undergoing pre-treatment assessment protocols. The authors did not state whether this was statistically significantly different from the number of tests ordered prior to implementation of the system. Interestingly, the largest increase in test orders (46%) was observed in the ‘other test’ category which consisted of special chemistry, serology, nuclear medicine and bacteriology tests, suggesting that more specialised diagnostic tests were requested more frequently after introduction of the CDSS. A summary of the number and type of tests ordered for patients undergoing assessment protocols pre- and post-implementation of the CDSS is displayed in Table 28.

In contrast to patients undergoing an assessment protocol, there was a 27% decrease observed in the number of tests requested per patient stay for those undergoing a transplant protocol, from 1047 pre-implementation of the system, to 768 post-implementation. This was most marked for the ‘other test’ category in which a 33% decrease in the number of tests ordered was observed. Again the authors did not state whether this decrease from baseline levels was statically significant. There was also a 44% decrease in the number of urgently requested tests per patient from 65 per patient prior to CDSS implementation to 36 post-implementation.

Summary of studies assessing the impact of the display of recommendations

The effect of the CDSS providing a recommendation was assessed in seven studies;32,50,97–100,104 two CRCTs,32,50 one RCT,97 and four pre–post studies. 98–100,104 Study quality was variable and only limited results were reported in one of the CRCTs by Hobbs and colleagues32 and in the RCT by Apkon and colleagues. 97

In the three studies that focused on the impact of the CDSS providing recommendations for the management of patients with hyperlipidemia, there was no significant effect in terms of increasing lipid test rates. 32,50,98 However, there was some limited impact in two, of increasing either the number of patients receiving a full lipid profile,32 or receiving an AST/ALT test. 50 However, to what degree these marginal increases would translate into improved management of patients with hyperlipidemia is unclear.

Likewise, the one RCT that assessed the impact of recommendations provided by a CDSS Coupler on the number of patient health-care opportunities fulfilled showed no significant benefit in terms of the number of either laboratory or diagnostic screening imaging tests undertaken compared with usual care. 97 In fact, there was a significant increase in laboratory test resource consumption compared with the usual care group.

In the one UPP study that assessed the impact of CDSS guideline recommendations conducted by Sanders and colleagues104 a small but significant reduction in the number of head or brain imaging studies was observed. However, despite this reduction compliance with the tests indicated by the guideline recommendations remained relatively low at only 60%. 104

Overall the results of the studies by Nightingale and colleagues99 and Boon-Falleur and colleagues100 conducted in specialist liver transplant centres showed that the implementation of guideline protocols for patient management may have differential effects according to the patient group. Overall, there tended to be a significant decrease in the number of laboratory tests ordered, the number of out of hours tests requested, and a reduction in laboratory costs. However, as appropriate levels of testing are driven by the patients’ condition and disease stage there were some increases observed in test requesting in certain patient groups. This was most pronounced in the study by Boon-Falleur and colleagues100 in which a 13% increase in the number of laboratory tests ordered in patients undergoing an initial assessment protocol was observed. A summary of the results from the studies assessing the impact of the display of recommendations is given in Table 29.

Search strategy

The generic search strategy used to identify relevant studies for inclusion in the three reviews is described in the section Identification of relevant studies, Chapter 3, and the search strategy documented in Appendix 1.

Study selection criteria

Apart from the study design criteria, the inclusion and exclusion criteria as outlined in Chapter 3 were identical to those for question two, the review of studies to assess the impact of CDSS in OCS versus OCS alone for diagnostic, monitoring or screening test ordering. For this review full CEA, CUA, CBA, CCA, and cost–comparison studies were eligible for inclusion. Economic evaluations that only reported the average cost-effectiveness ratios were also eligible for inclusion provided the incremental ratios could be calculated from the available published data. Based on the above inclusion/exclusion criteria, initial study selection was made on the basis of titles and abstracts from the search results by one reviewer, and a random 20% of these checked, unblinded by a second reviewer.

Data extraction strategy

Data were extracted by one reviewer and checked for accuracy by a second. Data extraction was limited by data availability in many studies. Relevant results were tabulated alongside the data for the main review to address the impact of CDSS in conjunction with OCS versus OCS alone (study question 2) and are presented in Appendix 3.

Quality assessment strategy

Both of the included identified studies primarily assessed the impact of CDSS plus OCS versus OCS alone, and were reported alongside evaluations of the impact of CDSS on process and patient outcomes. Therefore the reported cost comparisons were reported as secondary outcomes. Due to this the methodological quality of the studies was assessed according to the criteria for each study design outlined for question two, rather than by specific criteria for studies of economic evaluations. The methodological quality of both of the identified studies is previously discussed under the section including the review by Poley and colleagues96 (Chapter 5, Studies assessing the impact of the display of restricted lists) and the review by Bassa and colleagues98 (Chapter 5, Studies assessing the impact of the display of recommendations).

Results

As previously stated only two studies met the inclusion criteria, both of which were cost analyses. 96,98 A full description of both of the studies is reported in Chapter 5 in the sections Studies assessing the impact of the display of restricted lists and Study question 3. What features of CDSS are associated with clinician or patient acceptance of CDSS in order communication systems? on the impact of CDSS plus OCS versus OCS alone. One of the studies by Poley and colleagues96 was conducted in the Netherlands, while the second study by Bassa and colleagues98 was conducted in Spain. The perspective taken in both studies was the societal level. 96,98 The first study was a CPP study by Poley and colleagues96 conducted in 134 primary care practices including 234 primary care physicians (159 in the intervention group and 75 in the control group) which aimed to evaluate the cost analyses of a computer-based CDSS for ordering blood tests in a primary care setting compared with OCS alone. 96 The study consisted of a 6-month pre-intervention period and a 6-month post-intervention period. The CDSS comprised an optimal but restricted list of blood tests based on recommendations for blood test ordering for the patient’s indication, selected by the physician based on guidelines from the Dutch College of General Practitioners (http://nhg.artsennet.nl/). Minimum, maximum and base-case intervention costs (comprising the costs of both developing and installing the CDSS) were calculated. Development costs comprised: (1) personnel costs for reviewing 83 different guidelines for possible recommendations on blood tests, (2) writing the content, programming the software, testing prototypes, writing an explanatory leaflet about the CDSS, and writing instructions for installation and use. A summary of the minimum, maximum and base-case interventions costs is displayed in Table 30.

The minimum, base-case and maximum costs for installing the CDSS per practice were therefore €502, €670 and €839. As the CDSS was ultimately installed in 118 practices the total minimum and maximum estimate of the costs of developing and installing the CDSS was €41,000 and €48,000 respectively, with a base-case estimate of €44,000.

As the cost of laboratory requests depended on (1) the number of blood samples obtained and (2) the number and type of laboratory tests performed, data on the number of blood samples and blood tests performed in the pre-intervention and post-intervention phases were obtained from the laboratories. Costs were calculated by multiplying the number of blood samples and the number of tests by their unit costs. All unit costs were obtained from the national list of charges established by the Dutch Board for Health Care Tariffs (year 2003). The cost for obtaining a blood sample was set at €11.50, with the cost per test varying from €1.47 to €33.19 depending on the type of test. In addition to these costs (which included the cost of materials, laboratory personnel, and housing) the salary costs of a clinical chemist or medical microbiologist were included.

As previously stated in the review of studies on the impact of CDSS in OCS for diagnostic, screening or monitoring test ordering on process or patient outcomes, there was a significant decrease in the intervention group in the number of tests per order form (–6%) compared to the control group (+0%) (p = 0.001). This in combination with the type of laboratory blood test ordered and performed resulted in an insignificant mean cost decrease of 3% (€639) in the intervention group in the post-intervention phase, compared with a 2% (€208) increase in the control group (p = 0.09). Thus the CDSS yielded a mean cost saving of €847 per practice per 6 months (i.e. €639 plus €208).

The break-even point at which savings on laboratory costs exceeded the intervention costs was therefore reached after 5 months. Sensitivity analysis using the best-case scenario (upper limit of the 95% CI of the difference in laboratory cost requests; i.e. yearly savings of €3,669 per practice) and the minimum estimate of the intervention costs (€502 per practice) indicated intervention costs would be offset by savings as early as 2-months post-intervention implementation. Sensitivity analysis using the worst-case scenario (lower limit of the 95% CI of the difference in laboratory cost requests; i.e. increase of €282 per practice and the maximum estimate of the intervention costs (€838 per practice per year) indicated intervention costs would not be outweighed by savings on laboratory costs.

The second UPP study by Bassa and colleagues98 which assessed the impact on the effectiveness and costs of a practice guideline implemented through CDSS for the management of patients with hypercholesterolemia in a primary care setting, reported cost data pre- and post-implementation of the CDSS for pharmacological treatment, and laboratory tests. Only very minimal data were reported. The specific tests assessed were lipid profile and safety analyses (transaminases and muscular enzymes). These were costed using the Soikos database of health-care costs,127 with €0.46 for total cholesterol, €2 for LDL, €3 for HDL, €4 for triglycerides, €15 of CK, €1 for serum glutamic oxaloacetic transaminase, and €1 for serum glutamic pyruvic transaminase respectively. Despite the fact that there were no significant differences in the number of lipid profile tests conducted in the pre- and post-intervention periods (773 pre-intervention and 763 post-intervention, there was a significant increase in laboratory costs per patient from €41.8 per annum in the pre-implementation phase to €47.2 post-intervention [difference: +5.4 (95%: 2.0; 8.7) p = 0.0017]. The authors reported that this was due to a significantly higher number of safety analyses conducted in the post-intervention phase compared to the pre-intervention phase (803 compared with 734 respectively). However, overall patient treatment costs were reduced by a total of €78.4 per patient in the 1-year post-intervention phase mainly due to a decrease in the number of patients treated pharmacologically.

Study question 1: Which CDSS in OCS for test ordering are currently in use in the UK?

As stated in Chapter 4, the response rate from the survey of the 24 manufacturers and suppliers commissioned under the ASCC to provide CDSS support and functionality within the systems currently being deployed in the NHS was extremely disappointing, at 17%. The results have therefore been included in an Appendix rather than reported in the main body of the report as we do not consider them to be informative. Where any responses were received these were generally classified as being commercially sensitive data, and did little to elucidate which CDSS are currently either being trialled, deployed or implemented within the NHS.

Further contact with NHS CFH, the Healthcare Commission,15 NHS Purchasing Suppliers, and the NHS Supply Chain was made. However, as the contractual level is now managed at the individual SHA level, through the ASCC, no further useful information was gained as NHS Purchasing Suppliers and the NHS Supply Chain are not involved in this deployment.

Due to the time constraints of the assessment it was not possible to make contact with individual SHAs and PCTs to ascertain whether they are currently implementing CDSS and OCS as part of the NPfIT, and which systems if any they are implementing. It was therefore not possible within this assessment to ascertain which CDSS are currently being used within the NHS.

Study question 2: What is the impact of CDSS in OCS for diagnostic, screening or monitoring test ordering compared to OCS without CDSS on process and adverse events?

This section discusses the principal finding from the 24 studies that assessed the impact of CDSS in conjunction with OCS compared to OCS alone for test ordering. In the main body of the report the results from the studies have been discussed under the nominal categories according to predominant type of CDSS intervention(s). This section therefore follows the same format with the principal findings for (1) studies assessing the impact of the display of test costs (n = 3), (2) those assessing the impact of the display of previous test results (n = 2), (3) studies assessing the impact of reminders (n = 10), (4) studies assessing the effects of restricted test lists(n = 2), and (5) those assessing the impact of recommendations (n = 7) presented.

Study question 3. What features of CDSS are associated with clinician or patient acceptance of CDSS in OCS?

No studies were identified which met the inclusion criteria on the acceptability of CDSS to physicians or patients and therefore it was not possible to address this question within the context of this review.

Study question 4: What is the cost-effectiveness of CDSS in diagnostic, screening or monitoring test OCS compared to OCS without CDSS?

Only two studies met the inclusion criteria, both of which were cost-comparison analyses,96,98 although some limited cost data were also reported in the three trials that assessed the impact of the display of test charges by Tierney and collegues86 and the two RCTs by Bates and colleagues101 which are included in the review of the impact of CDSS plus OCS versus OCS alone for test ordering. None of these three trials met the inclusion criteria for the systematic review of economic evaluations of CDSS and were therefore not included in this section. Of the two included cost-comparison analyses the one by Poley and colleagues was conducted alongside a CPP study,96 and the one by Bassa and colleagues alongside a UPP study. 98 A full description of both of the studies is reported in Chapter 5, Studies assessing the impact of the display of recommendations and Study question 3. What features of CDSS are associated with clinician or patient acceptance of CDSS in order communications systems – on the impact of CDSS plus OCS versus OCS alone. Both studies found that the impact of CDSS plus OCS versus OCS alone had no significant impact on test costs.

Analyses conducted alongside the CPP study by Poley and colleagues found a mean cost decrease of 3% for blood tests orders (€639) in each of the intervention clinics compared with a 2% (€208) increase in the control clinics in test costs. However, this difference failed to reach conventional levels of statistical significance. 96 Likewise, the analysis conducted alongside the UPP study by Bassa and colleagues found a significant increase in the cost of laboratory tests (triglycerides, CK, serum glutamic oxaloacetic transaminase, and serum glutamic pyruvic transaminase) from €41.8 per patient per annum to €47.2 post implementation of the intervention. However, overall, patient treatment costs were reduced by a total of €78.4 per patient in the 1-year post-intervention phase, mainly due to a decrease in the number of patients treated pharmacologically.