The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials

Retropro

Retropro was a pragmatic point-of-care trial funded by the Wellcome Trust, randomising patients with high CVD risk to simvastatin or atorvastatin. Inclusion criteria included age over 40 years, a ≥ 20% or 10-year risk of developing CVD, primary hypercholesterolaemia and consent to participation. The primary objective was to evaluate the feasibility of conducting point-of-care trials using routinely collected data. The secondary objective was to measure laboratory and clinical outcomes in patients prescribed simvastatin or atorvastatin. CPRD was used to identify patients aged ≥ 40 years who had been registered for over 6 months with a practice. Patients with prior CVD or prescription of statins or recent history of liver disease or pregnancy were excluded. The CVD risks as predicted by QRISK®2 and Framingham risk score were then estimated. 33,34 The Framingham risk score uses an algorithm based on age, sex, ethnicity, smoking status, systolic blood pressure, ratio of cholesterol to high-density lipoprotein and history of diabetes mellitus and left ventricular hypertrophy. The QRISK2 score uses information on age, sex, ethnicity, Townsend score for socioeconomic status, body mass index, smoking status, systolic blood pressure, ratio of cholesterol to high-density lipoprotein, family CVD history, history of diabetes mellitus, treated hypertension, rheumatoid arthritis, chronic renal disease and atrial fibrillation. The Framingham risk scores were based on the publicly available algorithm. The Framingham predicted risks were adjusted by 1.4 for South Asians as recommended by the National Institute for Health and Care Excellence (NICE). 35 QRISK2 risk predictions were calculated using the commercial software program provided by CLINRISK Limited, using the 2012 version. Missing values for smoking status, systolic blood pressure and lipid levels and body mass index were imputed. 36 The statin daily dose was determined by the GPs based on clinical need or contraindications. End points of interest included repeat statin prescribing, death and incident CVD (as recorded in the EHRs) and the collection of a blood sample for genetic analysis. The blood sample was to be taken at the same time as the routine liver function test as recommended 3 months after starting a statin. The study team determined the trial topic in Retropro. This was based on the consideration that statins are widely used and on an assessment of NICE recommending research in the comparative effectiveness of different types of statins. 37 The target number for recruitment was 300 patients with, originally, a planned start date of August 2010 and end date of December 2012.

eLung

eLung was funded by the UK National Institute for Health Research (NIHR) Health Technology Assessment programme. It included patients aged ≥ 40 years with a medical history of COPD who, in the opinion of their GP, had an acute exacerbation of COPD with an increase of non-purulent sputum volume, who did not require immediate referral to specialist care for treatment of COPD exacerbation and consented to participation. Patients were randomised between immediate (prophylactic) versus deferred or non-use of antibiotics. The primary objective was to evaluate the feasibility of conducting randomised trials that use routinely collected data for follow-up. The secondary objective was to measure clinical and quality-of-life outcomes and forced expiratory volume in 1 second (FEV₁). CPRD was used to identify patients aged ≥ 40 years who had been registered for over 6 months with a practice and had a history of COPD [as defined by the Quality and Outcomes Framework (QOF) diagnostic Read codes38]. Patients were recruited during a consultation with their GP. Prescription of an antibiotic during the previous 2 weeks was an exclusion criterion (the protocol was amended from a period of 3 months). The choice of antibiotic was left to the GP, but they were advised to prescribe whichever antibiotic they would usually use as first line for acute bronchitis in a patient with similar characteristics. The rationale for not stipulating one particular antibiotic is that prescribing of antibiotic type shows marked regional and temporal variation. End points of interest included hospital admission for COPD exacerbation and prescribing of oral corticosteroids (as recorded in the EHRs). The original plan was for the GP to provide the participant with an electronic device with a daily measurement over 4 weeks of disease-specific symptoms and FEV₁ (eDiary; Vitalograph In2itive eDiary, 2012 version). During the conduct of the trial, participants were also offered the opportunity to complete by paper two quality-of-life questionnaires. These questionnaires were administered at baseline and at 4 weeks; both included the European Quality of Life-5 Dimensions-3 Level (EQ-5D-3L) and COPD Assessment Test questionnaires. The EQ-5D-3L provides a generic measure of health status over five domains (mobility, self-care, usual activities, pain/discomfort and anxiety/depression) with three levels of response (no problems, moderate problems or severe problems). The 4-week follow-up questionnaire also included a brief section regarding adverse events and any productivity loss in the previous 4 weeks. The funder of eLung had determined the trial topic of evaluating the effectiveness of antibiotics in COPD exacerbations and of measuring both clinical- and patient-reported end points. The target number for recruitment was 150 patients with, originally, a planned start date of August 2010 and end date of December 2012 (covering two winter seasons).

Overall design

In both trials, the GPs were expected to provide the general health care to the trial participants following their normal practice. The treatment allocation schedule was blinded to patients and clinicians, but they knew which treatment the patient was randomised to. The GPs had to confirm all inclusion and exclusion criteria and the consent of patients on the study website before randomisation.

Training of clinicians

Good Clinical Practice requires that every clinician involved in recruiting participants into a trial shall be qualified by education, training and experience to perform his or her tasks. The GCP inspectors commented:

Compliance of GPs may be an issue. Since there is an incentive for the GP to enrol subjects, then a busy GP investigator may not adhere to study procedure. Inappropriate delegation to other member of staff with no protocol or GCP training may lead to protocol and GCP non-compliance.

The study team addressed this by highlighting that the GP EHR systems are password controlled with a record of who entered the information. With respect to the key activities specifically done for point-of-care trials (i.e. patient recruitment and randomisation), these can be done only by GPs approved by the research team. In Retropro and eLung, the lead local investigator in each practice will need to complete web-based training on the protocol and relevant aspects of GCP. Other study-specific data may be collected by other staff members (e.g. nurses) on the web-based secure clinical trial management system (which is password controlled).

Consent process

Point-of-care trials can use different recruitment strategies including recruitment during an unscheduled consultation. The GCP inspectors commented that recruitment at the same consultation as the screening may be less acceptable as ‘Subjects should be given ample time to understand the implications of the study’. GCP inspectors also commented, ‘Continual consent may be required especially when the duration of follow-up is prolonged . . . subjects may have forgotten they are involved in a clinical trial’. In point-of-care trials, participants could ‘opt out’ of the trial by not redeeming the prescription and can return for a consultation. In order to address the concern of continual consent, the trial follow-up was divided into periods with active and passive follow-up. In the latter period, patients were followed only using the routinely collected anonymous medical data and this is considered an observational study. This approach is similar to that of the long-term follow-up of the West of Scotland Coronary Prevention Study. 41

Monitoring of study sites

Good Clinical Practice guidelines state that site visits should routinely be conducted (in order to initiate the site and collect data) and that central monitoring should be used only in exceptional circumstances. 42 The study team proposed the use of central monitoring without site visits. The GCP inspectors stated, ‘Considering a large number of sites are involved, central monitoring is insufficient to ensure GCP compliance at site level’. The study team responded:

Central monitoring may have superior power in detecting abnormalities (or serious GCP non-compliance) compared with site visits for studies that evaluate major clinical outcomes. A central assessment would be more efficient in picking up the poor quality as all the GPs data (outside the RCT) would be compared with that of others and as central monitoring would use data before and after the trial. A site visit could be useful for proxy outcomes measures that are measured in all patients, but it may be less efficient for outcomes that occur in only few patients.

The study team proposed that the trial populations would be monitored for differences with other patients in the EHR database. Also, it was agreed that a site visit could occur in the event of suspected fraud.

Data integrity

The trial data in the point-of-care trials consist of routinely collected data. The GCP inspectors stated:

There is no control on data quality either at the GP surgery or from third parties. These include incomplete data, entry error and omission. In addition, since data are not captured in a case report form, unstructured data are recorded by third parties.

The study team proposed to utilise the prospective, randomised, open, blinded, end point design. In this design, clinicians and patients are aware of the treatment allocation, but independent clinicians blinded to the randomised treatment adjudicate each end point of interest. 40 In case of any questions or discrepancies between the various data sources, a case validation approach could be used requesting the GP investigator to confirm the outcome.

Computer system validation

The trial data in point-of-care trials rely on electronic information systems. The GCP inspectors stated, ‘CPRD has no control on third parties software, data quality, interfaces and validation of databases. There is an inherent doubt on the quality of the data which cannot be qualified or quantified’. Furthermore, it was considered, ‘Third-party data providers are not legally involved in the study. The investigator should have formal agreements with these organisations to allow access by auditor when it is necessary.’ In conventional trials, GCP auditors typically have access to documentation outlining the testing of the information systems which often are developed specifically for the conduct of trials. In point-of-care trials, trial data are obtained from several computer systems, including the GP EHR systems, various hospital information systems that feed data into the national hospital registry and death certificate data. Numerous parties are involved in the recording and processing of these data, which are subject to detailed quality control by the NHS. It was agreed that the study team would seek and collect documentation on the main information systems used.

Verification of source data

The GCP inspectors commented, ‘There is a lack of source documents; data are directly entered in the database. It is not possible for an auditor to reconstruct reported data against source data’. In conventional trials, GCP typically compares the information in paper medical records (i.e. source data) with data in the trial database. The study team disagreed with the notion that electronic data do not constitute source data and that such data cannot be audited. Within several years, it is likely that most medical data will be collected electronically and it would be inefficient to maintain a system that would be based on transcription of electronic data to paper and then back from paper to electronic data. The GP EHR systems incorporate a full audit history of every record including time stamps.

Suspected side effects

Many countries require serious unexpected suspected adverse drug reactions (SUSARs) to be reported to the regulatory authority within 2 weeks of notification to the site. The original proposal consisted of requesting the GPs to expeditiously document any suspected ADR in the side effect fields of the EHR. The study team would review these events for reportability following a weekly electronic transfer of the EHR to the central research database. The GCP inspectors responded, ‘Weekly data extraction from sites to CPRD database might not meet the requirement of SI 2004 1031 Reg 33 that SUSARs are reported no later than 7 days after the sponsor was first aware of the reaction’. Following this discussion, two changes were made, at substantial cost, to the IT system. The first one concerned a change to daily transfer of the EHRs to the central database. The second change consisted of requesting GPs to also complete an ADR form on the study website.

Another concern of the GCP inspectors was, ‘The proposed protocol fails to meet the requirement to inform concerned investigators of any SUSARs which occurs in relation to the investigational product’. The study team tried to argue that the information provision should be similar to patients and clinicians inside or outside a point-of-care trial. Furthermore, SUSARs may include events that have already been reported to the regulatory authorities but did not warrant inclusion into the drug’s summary of product characteristics (e.g. their databases of reported suspected ADRs include many unlabelled events). The study team then proposed to report to regulatory authorities (and GP investigators) only SUSARs that were not listed in the summary of product characteristics and with less than five reports in the Yellow Card database. The GCP inspectors responded, ‘It is unacceptable to use such an imprecise definition of expectedness . . . it does not take into account of the population exposure . . . and there is no process to determine whether the Yellow Card submissions are relevant or not’. Following this discussion, the summary of product characteristics was being used to assess expectedness and procedures were implemented in order to report SUSARs to the regulatory authority, local NHS organisations and GP investigators.

Loss to follow-up

Study participants may leave the practice and be lost to follow-up. The point-of-care trials do not impose any restrictions on the use of the study medication if a patient leaves the practice. The GCP inspectors commented, ‘Some contingency should be defined for patients transferred from CPRD centres to non-CPRD centres. It is unclear how patient’s ADRs are captured in this situation’. Loss to follow-up occurs in all long-term trials. The ability to link various health-care data sets will ensure that outcomes that lead to hospitalisation or death can be captured for trial participants, even after the participant has left the practice (unless the patient has emigrated).

Drug accountability

Drug accountability allows the reconstruction of the trial and documents what medication was received by the site and what medication was received by the patient. 42 In point-of-care trials, the drugs will be provided by any UK pharmacy based on the routine process for dispensing licensed medicines following a prescription by a clinician. The GCP inspectors considered, ‘The lack of drug accountability may not be a problem, but it is not possible to monitor compliance’. The study team responded by stating that prescription records do allow an assessment of compliance (for long-term therapies) by estimating the medication-to-possession ratio. This ratio is the number of days covered by the prescribed therapy divided by the total period of follow-up. This is routinely done in studies that use health-care records. However, this method does not measure whether or not the medication was actually taken.

Fraud detection

One part of fraud prevention in point-of-care trials will be to statistically monitor for unusual data patterns. Currently, site monitoring is seen as the most important approach to fraud detection. The GCP inspectors stated, ‘The statistical method for detecting fraud needs to be validated to ensure the methodology is sufficiently sensitive to identify data irregularities’. The study team replied that there is indeed no evidence to substantiate the sensitivity of statistical monitoring for fraud detection, but such evidence is also lacking for the effectiveness of site visits. Site visits may not be effective in detecting fraud for studies that measure only major clinical outcomes as most sites will not have any records of the end point of interest. Furthermore, trials within existing EHR research databases may minimise the risks of fraud as each GP is not able to recruit ‘fake’ patients (only patients permanently registered for at least 6 months will be eligible), and as the data for the participants prior to and after the trial can be used for fraud detection. The open access of the EHRs to other health-care professionals in the same practice may also minimise false data entry.

Conclusions of the Good Clinical Practice review

The general concluding comments of the GCP review were:

The regulatory environment has moved on . . . The Risk Adaptive model, launched in April 2011, has proposed a reduction in the level of monitoring and pharmacovigilance requirements for the lower-risk studies. This model is well suited for the pragmatic randomised trial proposed by the CPRD . . . It is recommended that CPRD should risk assess its trials and categorises them based on their risk and included in the Clinical Trial Authorisation Applications. Once approved by the Clinical Trial Unit, Category A studies would be deemed as lower risk.

Barriers

Time was perceived to be the biggest barrier to recruiting patients into trials. Time seemed to be a universal barrier in that all 13 participants mentioned time at least once:

I think probably the main challenge is time.

AGP-1

I’m someone who finds it hard ever to run to ten minutes . . . even just covering what I’m trying to do now . . . my average is 13 or 14, if I had another five minutes, that just wouldn’t work over the course of a three hour surgery, I’d be three hours late.

NAGP-5

Some participants discussed ways in which the barrier of time could be overcome. Examples included the establishment of dedicated research clinics for recruitment, additional blank appointments to allow for delays, protected research time, and/or lengthening consultations. Others felt that clinicians could merely refer potentially eligible patients to researchers for recruitment, thereby minimising the impact of time on the clinician. In addition, a number of GPs talked about how money could be used to ‘create’ more time:

It might be that the money is used to increase time for the other team members who are involved. Like if they use the money to have extra appointments or blank slots . . . that’s fine . . . So I think it’s a practical thing because it’s whether the money can be used to make the thing happen.

AGP-1

Yeah, but that’s still, when I mean committing more time, that’s still practice time . . . and . . . creating extra appointments purely for people to come back and discuss taking part in research feels like quite a generous use of time. There might need to be some money involved . . . You know, people’s time is money, sadly.

NAGP-2

Another GP thought financial incentives would compensate only to a limited degree as, ultimately, time is of short supply. This implies that, regardless of financial rewards, the time taken to recruit patients still needs to be minimised.

Anything that takes longer than 30 seconds . . . If this would be just as long as choose and book, I’m sure GP’s would say no I’m not bothered. Because it just takes too long. Even if the financial incentive would be there.

NAGP-1

Interestingly, although all four nurses discussed time as a significant barrier, none mentioned using financial incentives to overcome time. Instead the nurses appeared to focus on minimising the impact of time by recruiting patients during dedicated clinics.

Six participants alluded to the fact they may be liable to select patients. Not offering the trial equitably to all potentially eligible patients could introduce selection bias and limit the trial’s external validity:

I think in a GP setting I would worry that I wasn’t being completely random.

NAGP-4

It was commonly reported that a non-random selection of patients was a way of protecting potentially vulnerable patients who, for example, lacked understanding or for whom it was thought a trial would be too burdensome due to pre-existing comorbidities.

The second biggest reported barrier after time was GP concerns about patients not understanding the information:

I suppose it’s whether they can understand, you know, how many of them will be able to truly understand and, you know, give informed consent to a trial . . . That’s quite an issue. And I think of the patients I see, I think some of them it will be quite difficult to explain a trial to them and to understand what it really meant . . . Just lots of, you know, language, literacy skills, you know, a whole host of things.

AGP-2

Some GPs felt that patients’ cultural beliefs may prevent them from participating in a trial:

I mean particularly in an area like this, you’ve got language, cultural . . . issues as well to think about. You know, what’s acceptable and what isn’t.

PN-2

As previously stated, these barriers (culture, understanding and language) may be compounded by selection bias:

There are big problems with language in inner-city general practice. And again, that’s going to deter you because you know any research study’s going to be a battery of questionnaires, you’ve got to be pretty literate to go through most of those questionnaires, at least certainly in the mental health studies. And then who do you recruit to your studies? Surprise, surprise, it’s the middle class, literate people.

AGP-3

The two biggest barriers (time and understanding) were also found to interact in that, for some participants, insufficient time meant that they did not feel they would be able to adequately explain a trial to patients who had difficulty understanding:

We should have sufficient time to create that level of awareness and understanding.

PN-1

Facilitators

Clinicians reported being more inclined to recruit patients if they perceived either a potential benefit to their own patients or to the wider population:

I think people might find it difficult to ask people do they want to be the subject of research. Unless it’s offering them a benefit.

AGP-2

Well we’re sort of doing research perhaps a bit for our own interest, perhaps a bit for our career promotion, but actually it’s because of the greater good for mankind.

AGP-3

The importance of perceiving a benefit also highlighted issues of clinical uncertainty; that there is no known difference in effectiveness between trial treatment interventions. Three GPs did demonstrate an understanding that uncertainty had to exist for a trial to be deemed ethical. However, despite this, some participants described feeling uneasy:

If it was a randomised trial and some people might not get treatment . . . I think that would be quite difficult . . . I think it would conflict with your clinical role of doing the best you could for that particular patient.

AGP-2

Many organisational factors thought to facilitate recruitment were concerned with separating research and clinical roles. These included establishing dedicated research recruitment clinics; employing someone purely to recruit; and having a separate research team solely responsible for recruiting. Other organisational factors that could facilitate recruitment were concerned with making the process easier and ensuring adequate support systems were in place. Some felt that it was important that advertising or promotion of a trial was conducted in order to facilitate recruitment.

Modifying factors

Nine participants talked about seeking informed consent from patients for trial participation. Interestingly, there appeared to be a difference between GPs and nurses. Specifically, most GPs felt that consent would be too difficult to obtain within a consultation and/or they were unwilling to obtain consent from patients:

No, no, no, no, no. That should be done by the study group. In fact the whole consenting and giving more information should be done by the research team.

NAGP-1

Reasons for concern included a lack of research experience, feeling it would be unethical to ask patients for an immediate decision, and insufficient time. Two out of four nurses discussed consent and neither thought that this was an issue, with one stating it should be easy. The nurses appeared to liken it to taking consent in other routine clinical situations. GPs seek consent from patients routinely within their clinical practice but yet felt there was something different about taking informed consent for trial participation which seemed not just due to insufficient time. The unwillingness to seek consent was linked with a feeling that it was something the researchers, and not clinicians, should be doing.

Financial constraints were mentioned by eight participants. Some GPs felt that financial incentives for clinicians and/or practices would be important in facilitating recruitment. Some thought that financial reimbursement would be required to offset additional costs, and may be an important motivator, or at the very least mitigate logistical problems. Interestingly, only one nurse mentioned financial incentives as a facilitator of recruitment.

Ten participants felt that forgetting could impede recruitment and/or reminders could facilitate recruitment. All the GPs mentioned memory as important, in contrast to only one nurse. However, one GP did comment that forgetting was by no means the biggest barrier. Others also mentioned how heightening personal involvement could prevent forgetting:

Because I think if I’ve got an interest in it, I’m more likely to think of it.

NAGP-6

Concern for the clinician–patient relationship was recognised as a potential issue by some participants. However, only one GP was worried about a coercive role which could prevent them from recruiting:

I would hate to feel . . . that my patients felt obliged in any way because I was asking them because I think you have quite a powerful influence as a GP on what patients do or don’t do and I think they find it quite hard to say no to their own GP . . . it’s just a feeling . . . it’s not based on experience, really . . . But I do think . . . if I were sitting in a room with a patient saying . . . etc. etc. . . . I’d worry that that’s a bit of a coercive role to be playing.

NAGP-2

In contrast, clinicians seemed to be aware of the fact their relationship with patients may actually facilitate recruitment, and this appeared to be universal among GPs and nurses:

I mean, the advantage is that they have a relationship with us, they trust us, so that’s presumably why the researchers want to get us involved . . . it might help recruitment.

NAGP-3

I think it’s about the relationship between me and the patient . . . the new patient, perhaps they might be a bit more hesitant, but if it’s a patient that I’ve been seeing and developed a relationship with over a long time, and they have confidence in me . . . I think they’d be more willing to participate in something that I recommend to them.

PN-4

Eleven participants felt that having a research question of personal importance or interest to them was the key to facilitating recruitment. Equally, an uninteresting or unimportant topic would act as a barrier to recruitment. Ownership seemed to be particularly important. Linked with ownership, clinicians reported the importance of personally understanding the trial, as well as having experience and knowledge of recruiting to trials generally. Interestingly, this seemed to be more significant for all four nurses. For GPs, knowledge and understanding was associated with being able to seek consent appropriately. For nurses, it was about having adequate (ideally face-to-face) training.

Attitudes towards computerised trial recruitment

Just under half the participants felt the use of CTR tools would be a good thing, with eight participants liking the idea of using prompts and reminders during consultations. One GP and one nurse also thought CTR tools could add validity to recruitment during consultations:

I think people . . . know that we are always using the tools on the computer, so I think . . . computerised tools as a part of their consultation . . . will be more effective and it will sound more real to the patient.

PN-1

Negative attitudes towards CTR tools were centred on concerns that pop-ups could disrupt the consultation, although others felt this would not be the case given how used primary care clinicians are to interacting with EHR and QOF alert pop-ups during consultations.

Two participants were unsure how they felt about CTR tools and two GPs expressed overtly ambivalent views. Perhaps surprisingly, only one GP expressed concern for patient confidentiality.

Attitudes towards Local Eligible Patient Identification Service

Just over half of the participants liked LEPIS. One GP had neutral feelings and two participants stated that their attitude depended on their motivation or the amount of time they had. The interface was largely seen as user-friendly and easy to use, although two GPs thought that the icons were too small or similar, and two GPs felt that LEPIS was too simplistic. Two GPs liked the fact that LEPIS could be ignored, although one GP saw this as a potential design flaw as LEPIS could be ignored consistently. Some participants liked the fact there were different options to choose within the consultation, but one GP felt that having so many options could be confusing for the clinician.

Four participants felt that LEPIS was too slow with too many steps. Five GPs felt that having to log on would be burdensome, take too much time and be prone to forgetting passwords. Interestingly, this was not a concern for nurses who saw the requirement for logging on as important to protect patient confidentiality. This disparity between GPs and nurses may reflect the fact the latter have longer consultation times. In addition, four GPs felt that a direct link to patient information sheets would be required rather than additionally having to log on.

Feasibility of proposed Retropro and eLung trials

The consensus was that complete trial recruitment within a consultation would not be feasible and that patients would probably need the opportunity to think about participation outside consultation:

I think that would be a bit too quick . . . And I think you’d want the patient to go away and read about it and think about it . . . mean it wouldn’t be ethical to do it like that.

AGP-2

Retropro was felt by some to fit nicely with routine patient consultations:

Well I mean it is only a statin and I’m quite often trying to coerce patients into not taking their expensive atorvastatin and having the cheaper simvastatin so maybe actually it would be rather easier to say we’re really trying to find out which one’s better . . . and maybe have some evidence and what about being randomised between the two? . . . But I’d still worry about having enough time even to do that on top of everything else.

NAGP-2

In addition, there were fewer concerns surrounding Retropro because patients could have the opportunity to go away and think about participating prior to making a decision. In contrast, eLung gave more cause for concern as patients would present acutely unwell, rendering them more vulnerable and less able to give fully informed consent, and in more time-pressured emergency consultations:

If someone’s that ill with COPD exacerbation maybe they’re not mentally . . . in the right place to decide whether they need antibiotics or not.

NAGP-3

As a result, eLung was generally felt to be less feasible than Retropro. However, the pragmatic nature of the proposed trials, particularly Retropro, led to slightly less hesitancy and more positive attitudes from clinicians:

Research which is sort of GP friendly, it’s stuff that we deal with that’s relevant to us . . . and it’s medical knowledge that is our territory . . . so therefore that isn’t too bad.

NAGP-3

The key findings of this qualitative study of views on CTR were as follows:

1. Time was the biggest barrier to recruitment.

2. Financial incentives could be used to minimise the impact of time; financial incentives may motivate personal involvement; financial incentives appear less important to practice nurses.

3. Understanding (or lack of) was the second biggest barrier to recruitment.

4. Selection bias (linked with understanding, language barriers, culture and vulnerability) acts as a barrier to recruitment.

5. Presence of clinical uncertainty would facilitate recruitment; personal involvement and interest were important facilitators.

6. Informed consent acted as a modifying factor; nurses were more amenable to seeking consent.

7. Concern for the clinician–patient relationship could act as a barrier, but most felt it would facilitate recruitment.

8. Attitudes towards CTR tools were favourable.

9. Attitudes towards LEPIS were generally favourable with positive feedback.

10. Point-of-care trials seemed feasible; Retropro is easier to conduct than eLung.

Recruitment into trials

In line with literature, time was identified as the prominent barrier for GPs and nurses to recruiting patients into trials. Many of the other barriers, facilitating factors and/or modifiers identified within this study have also been identified in prior literature. However, some important differences and novel themes emerged, perhaps because this study examined attitudes towards clinicians recruiting patients themselves rather than, as with the majority of studies to date, clinicians determining eligibility and then referring patients to researchers for recruitment.

For example, it has previously been demonstrated that seeking informed consent is a barrier to recruitment. 53,61,62 However, this study identified a difference between nurses and GPs in that consent was a barrier for the latter but not the former. Similarly, all 10 nurses interviewed by Potter and Dale59 felt comfortable seeking consent from patients for participation in a trial, even though only four had done so previously. 59 This discrepancy between nurses and GPs requires further research, but may suggest nurses could play more of a role in recruiting patients into trials than has traditionally been the case.

One reason gaining informed consent may be problematic is that:

It may not be easy to forsake the role of the confident prescriber and adopt the attitude of the researcher who is uncertain about the most effective treatment available.

Van der Windt et al. 63

This relates to clinical uncertainty and, as with previous studies,63–67 this was thought (if present) to facilitate recruitment. Thus, providing clinicians with information to demonstrate the need for a trial may aid recruitment. 64,68 The negative impact of a lack of clinical uncertainty was demonstrated by concerns expressed regarding eLung – participants were reluctant to randomise patients to no antibiotics even though no convincing evidence for antibiotic efficacy exists when patients have non-purulent sputum. Similar unease was found in a study investigating the management of urinary tract infections without antibiotics. 65

Concern for the doctor (or clinician)–patient relationship has frequently been cited as an important barrier. 52,53,61,62,67,69 In contrast, such concern was not demonstrated in this study and, in fact, it was thought that such a relationship might facilitate recruitment. Some authors have previously suggested that the existence of a relationship, and thus understanding and trust, may aid trial recruitment. 58,70,71 The results of this study support the importance of the clinician–patient relationship in helping trial recruitment.

Computerised trial recruitment and Local Eligible Patient Identification Service

Participants generally reported favourable attitudes towards CTR, with LEPIS largely perceived to be user-friendly and intuitive. 72 The fact that primary care consultations already utilise computer-based prompts and pop-ups was taken by some to demonstrate that further prompts would be excessive, although others saw this as positive, in that they were already an accepted part of consultations. Overall, as long as issues such as speed and ease of use are dealt with, it would seem that primary care clinicians are, in principle, amenable to the use of CTR tools.

Methodological limitations

Conclusions of this study may be limited because of the small sample size. However, emergent themes were identified and data saturation has previously been demonstrated with similar numbers. 73 Initially it was intended to obtain a stratified purposive sample of academic versus non-academic GPs and nurses to determine if these subgroups held disparate beliefs and attitudes, and to assist comparisons within and between groups. 74 Given resource and time constraints, a convenience purposive sampling technique was used. However, only four practice nurses were included within this final sample, which will not be sufficient for data saturation, although many recurrent themes did emerge. This methodological limitation was recognised and efforts were made to minimise any negative impact by ensuring a range of primary care clinicians were interviewed. No participants worked within a practice that currently employs Vision EHR. This should have limited impact as the early LEPIS prototype had not been integrated with an EHR and the principles demonstrated should be equivalent regardless of EHR. The participants worked within inner-city, deprived and ethnically diverse areas and the identified barriers may have been unique to local concerns – the latter was probably reflected by the frequency with which participants mentioned language barriers.

Conclusion

As with previous literature, insufficient time was perceived to be the biggest barrier to recruitment. Lack of clinical uncertainty was also a significant barrier, which could be easily remedied by providing clinicians with information pertaining to the importance of a trial. The fact that nurses appeared more comfortable than GPs in seeking informed consent for trial participation requires further investigation, but demonstrates that nurses may be underutilised in trial recruitment. In contrast to previous literature, the clinician–patient relationship was perceived to potentially facilitate recruitment. Employing clinicians to recruit patients could improve trial recruitment rates, and could be further aided by the use of CTR tools to enable such recruitment to be embedded within consultations. The fact that primary care clinicians held generally positive attitudes towards the use of CTR tools and LEPIS in particular, suggests that recruitment within consultations is feasible. Indeed, the pragmatic nature of the proposed point-of-care trials appeared reassuring to many of the clinicians, although trials which did not include patients with acute illness were favoured. In order to further aid understanding of trial recruitment, qualitative analyses should be embedded within proposed point-of-care trials.

Electronic health record system

There are a number of EHR systems currently used by UK general practices (including EMIS, System One TPP and Vision). CPRD was working only with the Vision system at the time of this project. The clinicians use these system to enter administrative, clinical and prescription data for patients (see process 16 in Figure 1 ). The Vision platform, managed by In Practice System is enabled to anonymise and download patient data, and send incremental data collections to CPRD on a regular basis (every 4 weeks). As part of this project, a daily automated transfer of EHR was implemented between participating study sites and CPRD (see process 1 in Figure 1 ). The Vision system also generates an extract file in real time, which is readable by the flagging technical LEPIS (see below).

Local Eligible Patient Identification Service

Local Eligible Patient Identification Service is a programme developed by the study team residing on clinicians’ computers. It receives patient information from the local extract file (see process 17 in Figure 1 ) and interacts with an external server [central control service (CCS)] in order to download the list of active studies, authorised staff, and the fully encrypted patient eligibility lists (see process 15 in Figure 1 ). It uses this information to compare patients against the eligibility criteria of active clinical trials to find suitable participants. If a patient is found to be (potentially) eligible, a graphical pop-up is generated on the clinician’s computer screen (see process 18 in Figure 1 ). If the clinician decides that the patient is eligible, LEPIS triggers a web link for accessing the study website. This link automatically initiates the local web browser with the unique webpage for the study and the hashed (encrypted) patient identifier is then sent to the study website (see process 10 in Figure 1 ). The clinician can then log onto the study website and confirm the patient’s eligibility. The website will then retrieve the randomisation result and inform the clinician of the outcome. LEPIS can be configured to pop-up only for clinicians authorised take part in the study. Figure 2 shows the LEPIS pop-up boxes in Retropro and eLung. The following buttons are present in the LEPIS pop-up box:

• More information: this opens a new window with further information about the trial.

• Recruit patient: this opens a webpage, which allows the patient to be recruited in real time.

• Patient wants time to think: this will set the notification to be generated again when the patient next returns to the clinic.

• Patient not interested: this will set the patient as not interested in the specific trial; the practitioner will not be asked about recruiting the patient to this trial again.

• Patient not eligible: this will set the patient as being not eligible (i.e. the automated eligibility check was inaccurate); the practitioner will not be asked about recruiting the patient to this trial again.

• Do not disturb me for 2 hours: this will temporarily deactivate LEPIS for 2 hours; no notifications will be generated during that time.

• Close for this patient: this will close the window; this can be done with or without first registering a response.

• Other actions: this is a list of other actions that can be performed. Currently, this consists of only one action.

  1. Patient not interested in ever participating in trials: no notification will ever be generated for this patient again.

FIGURE 2.

Local Eligible Patient Identification Service pop-up boxes for (a) Retropro; and (b) eLung.

Extraction file

As soon as a new consultation is initiated by a health-care professional (nurse or doctor), a text file (extract file) is created by Vision (see process 17 in Figure 1 ). This file holds limited information for the patient currently in consultation. To ensure that this information cannot compromise patient privacy, no identifiable information is stored. Once it has been read by LEPIS, it is automatically deleted. This file is held in the local file system and acts as a bridge between the EHR system and LEPIS. This extract file will not be created for clinicians who have opted-out from LEPIS and/or for practices that have not activated LEPIS within the EHR system.

The vendor of the Vision system proposed this process of LEPIS searching an extraction file, as this would not adversely impact the performance of Vision. Complex searches can be done centrally with the extraction file providing the latest EHR information which can be analysed by LEPIS. This extraction file can inform LEPIS of a patient’s record being in consultation or a modification to the patient’s record taking place (e.g. a new Read code being added). Through this, LEPIS operates independently of the EHR.

Central control service

The CCS is a central management service for handling the construction and distribution of study information to LEPIS. The lists of eligible patients (see process 14 in Figure 1 ) and authorised clinicians (see process 13 in Figure 1 ) are updated daily and sent through an automatic process to this repository, which LEPIS uses to compare in real time with the patient and staff identifiers at the time of the consultation. The CCS can be considered as a bridge between the EHR research database and LEPIS at the practice level. The CCS also holds information about the study and limited non-confidential details about eligible and recruited patients (stored in an encrypted manner). Furthermore, connections between LEPIS and the CCS are encrypted using Secure Sockets Layer (SSL) and only hosts within the national broadband network for the NHS (N3) are authorised to access CCS. The lists of eligible patients and staff and study details are sent daily to the clinician’s desktop for use by LEPIS (see process 15 in Figure 1 ). LEPIS also sends information regularly to the CCS on the flagging and actions taken following flagging at each clinic (see processes 21 and 22 in Figure 1 ).

Study website

This is a dedicated website used to recruit patients, conduct the randomisation and collect data on/from recruited patients. The use of the secure pages of the study website is available only to authorised users; therefore, registration is mandatory followed by an approval after further checks before the site can be used. The study website allows a user to securely log in and recruit eligible patients to the relevant study or record further information. Clinicians can either access the study website through LEPIS or directly access the website by typing in the web address. In the latter situation, the clinician needs to provide the patient identifier as stored in the EHR, year of birth and sex. The study website will then check whether or nor this patient identifier is recorded in the study database and if the year of birth and sex match with the information in the EHR research database (see process 22 in Figure 1 ).

The study website also acts as the repository for study information (such as the protocol and consent forms). Clinicians can also access the study website in order to submit study-specific data, such as the ADR form (see process 19 in Figure 1 ). There is also the possibility to record that the patient wishes to opt out of further data collections (i.e. the future EHR information will not be processed for the study).

Study database

The study database is a central management service for handling all the forms submitted through the study website. It is the main storage of all data used by the study website, such as registered users, practices approved to recruit, available studies, eligibility lists of patients, lists of approved staff, maximum number of patients who can be recruited at each site and administrative management of the website (including details of who accessed the study website at what time and records of any changes to forms). The study database is also the main source data for internal analysis regarding the various studies. Information on recruited patients (see process 12 in Figure 1 ) and approved staff (see process 11 in Figure 1 ) is shared daily between the study website and the study eligibility checker.

Data processing system

The data processing system is responsible for managing and processing EHR collections received from participating trial practices on a daily basis. Practice data are processed to generate lists of patients eligible for each trial. The system consists of four main components: data collection manager, data processor, trial eligibility checker and patients processor. A brief description of each is provided below.

Handling of suspected adverse drug reactions

Clinicians can enter data into the side effect fields within Vision or use a medical code to represent a suspected ADR. The latest EHR information is checked daily for both kinds of entries for recruited patients. The trial side effect database (see process 28 in Figure 1 ) is populated if a potential ADR is detected and an ADR report generated containing details of the potential ADR and the patient’s medical and prescription history for the previous 6 months. This report is then immediately sent to the principal investigator (see process 29 in Figure 1 ) by e-mail for review. Trial-specific guidelines stated that a clinician should fill and submit a suspected ADR form on the study website. Forms submitted on the website are processed daily (see process 24 in Figure 1 ) and an e-mail is sent to the principal investigator (see process27 in Figure 1 ). The ADRs are reviewed by the principal investigator for reportability to the regulatory authority. Suspected ADRs as entered in free text into the EHR were not analysed because of the variability in the recording of this information.

Regulatory electronic submission of serious unexpected suspected adverse drug reactions

If an ADR is considered serious and unexpected, this report is then submitted electronically to the regulatory authority (see process 31 in Figure 1 ). The study website uses a specially designed tool which communicates with the electronic SUSAR reporting system of the UK regulatory authority eliminating the need for further manual input or manipulation of existing data. Log files are maintained of this reportability assessment (see process 30 in Figure 1 ).

Data monitoring during study

The EHR information on recruited patients (see process 32 in Figure 1 ) can be used for monthly analyses of the characteristics of recruited patients and comparisons with patients not recruited into the study. Fraud detection and evaluation of data irregularities can also use this information (see process 33 in Figure 1 ).

Performance of the information technology system

Before rolling out the IT system, the complete IT system was extensively tested on development servers with a replica of data. We also identified a few practices, installed the software and carried out tests with GPs (although this was not as extensive as intended due to the short timelines). A software company also tested LEPIS in terms of performance, scalability and security. The biggest challenge with the roll-out of the IT system was the limited time which was allowed for the testing given the substantial delays in obtaining research approvals. The initial plan was to implement and test the IT system in a few practices locally, but this was replaced by a larger and quicker roll-out across the country in order to start patient recruitment. Table 6 illustrates the challenges in the implementation of the software system. Updates in Vision were required for LEPIS and the daily downloads of EHRs followed by reconfiguration of Vision at the practice. Although this process was not complex, some practices were behind in uploading Vision updates. The implementation of LEPIS provided several major challenges. In some practices, staff were not authorised to load software and detailed negotiations with third parties were required. The network set-up, network restrictions and the clinician’s computer specifications varied between practices. One of the issues encountered was low performance of Vision after LEPIS installation. This was found to be related to the low memory space on local desktops and LEPIS loading of eligibility lists into the memory. This was encountered when starting a new study that required much larger eligibility lists (i.e. all patients in a practice). This was addressed by reconfiguring the patient eligibility lists and respective parts of LEPIS in order to handle larger eligibility lists with small memory usage. LEPIS was also modified in order to provide more diagnostic tools including tests of the network status. Log files could also be sent automatically during a limited period of time to the CCS rather than the practice sending the log files. Even after a successful instalment and test, LEPIS sometimes did not pop-up at the time of consultation in some practices. It was established that LEPIS generally functioned but that the notification did not occur due to a step being inadvertently missed or procedures not being followed (e.g. the extraction file generation was not enabled within Vision, the right Read code or description was not entered in the new consultation, or the desktop had not been rebooted after LEPIS installation).

The initial approach by the study team was to send practices a DVD with the flagging software and installation instructions. This was later simplified by sending links for the software installation. However, many practices were unable to load LEPIS successfully, partly because installation steps were not followed and partly because of technical challenges. The study team visited several practices to provide support in loading and testing LEPIS (typically, this was done in one visit). The study team explored the possibility that support in loading and testing LEPIS could be provided by the vendor of Vision, but this was not adopted given that most of the installation challenges were related to settings of local desktops not controlled by the vendor of Vision. Although the site visits were laborious, the study team did consider them to be very useful for practices that had encountered issues in loading LEPIS. Once LEPIS was installed successfully, further visits were not needed.

No major issues were encountered in the development and implementation of the systems other than the flagging software. The study website allowed staff registration and completion of the recruitment, ADR and opt-out forms. Daily and monthly processing of eligibility, the automated identification and e-mailing of suspected ADRs and reporting of recruitment numbers were also implemented successfully. An issue was the perceived disagreement in the assessment of patient eligibility between the clinician and the flagging software. Clinicians were expecting a LEPIS pop-up or wanted to recruit directly on the website but the patient was not found on the website and/or LEPIS did not pop up. In some instances, this was related to EHR information not being noted by the clinician (e.g. a patient received statins previously many years ago, which was an exclusion criterion for Retropro). In other instances, this was related to differences in the eligibility assessment (e.g. differences in the estimates of CVD risk owing to incomplete EHRs). However, it was possible for patients to be added on an ad hoc basis to the eligibility list.

Retropro

Figure 3 shows cumulative patient recruitment over time in Retropro. It completed recruitment in November 2013 (301 patients). The 301 Retropro participants were recruited in 17 practices by 25 GPs. One practice recruited 75 patients, nine practices between 10 and 50 patients and seven practices each fewer than 10 patients. Retropro started in September 2012 and recruited steadily over time. Retropro recruited 20.6% of all statin starters in recruiting practices (this number of statin starters included patients with a 10-year CVD risk of < 20% and patients seen by non-trial GPs in the practices). The recruitment rate of Retropro was 1.1% of the statin starters in the EHR database. Several recruitment strategies were used in both trials. Patients could be recruited either through LEPIS flagging during consultation (with provision of a link) or through direct access of the trial website. The criteria for LEPIS flagging were changed over time. Flagging initially occurred in both trials at EHR entry of specific clinical codes and later flagging occurred at start of consultation. Practices varied in their interest in using flagging for recruitment and the preferred criteria for flagging. A challenge in Retropro recruitment was the discrepancy in results between the CVD risk assessment done centrally and that done by the GP. Prediction of high CVD risk varied considerably between the different risk scores. In addition, some GPs considered risk factors not considered by the risk scores. As a consequence, some GPs wanted to recruit while not meeting the CVD threshold according to the assessment done centrally (e.g. one patient had a 12% CVD risk according to the Framingham risk score based on the EHRs).

FIGURE 3.

Cumulative numbers of patients recruited into (a) Retropro; and (b) eLung.

eLung

eLung recruited 31 patients (out of an intended 150). The original plan for eLung was to recruit over two winter seasons, but this could not be done because of the cessation of funding and long delays in study initiation (first patient was recruited in February 2013). The 31 eLung participants were recruited in six practices by seven GPs. One practice recruited 21 patients, whereas the other practices recruited each fewer than five patients.

Retropro

Table 9 shows the characteristics of the Retropro participants and of non-trial patients who started statin treatment in recruiting and non-recruiting practices. The 10-year CVD risks were found to be higher inRetropro participants than in non-participants (10-year CVD risk was an inclusion criterion). Furthermore, Retropro participants were less likely to have a history of diabetes mellitus than statin starters in the same practices (8.6% and 21.1% respectively). The mean number of patients aged ≥ 40 years was 4981 in thepractices recruiting for Retropro and 4633 in non-recruiting practices. The percentages of patients who had received a statin prescription were comparable in these practices (22.0% and 21.9% respectively). The random treatment allocation was followed by the GP for all participants, except for nine participants who did not have a baseline statin prescription in their EHR (six of these had been allocated to simvastatin and three to atorvastatin). The most frequent daily dose was 40 mg for simvastatin (71.3%) and 10 mg for atorvastatin (58.3%). A blood sample for genetic analyses was provided by 234 participants; this corresponds to a collection rate of 88.6% (at the time of this report). The EHR research database was reviewed monthly for the characteristics of recruited patients compared with non-recruited patients. A monthly analysis reviewed the recording of major clinical outcomes (such as heart attacks) comparing the trial and non-trial populations.

eLung

Table 10 shows the characteristics of the eLung participants and of the potentially eligible non-participants in recruiting and non-recruiting practices. eLung participants received on average more oral corticosteroids in the 6 months before starting the trial compared with the non-trial participants. Furthermore, the rate of prescribing in the year before was considerably higher in eLung participants (73.2%) compared with other COPD exacerbation in the same practices (28.1%). eLung recruited 32.3% of all potentially eligible patients in the recruiting practices (this number includes patients seen by other GPs); the best recruiting practices enrolled over half of their potentially eligible patients (55.2%). There were 3518 potentially eligible patients in all CPRD practices; 0.9% of these were recruited into eLung. The random treatment allocation was followed by the GP for all 31 participants; 11 patients were allocated to immediate antibiotics and 20 to deferred or non-use. The type of antibiotic was amoxicillin for seven, doxycycline for three and clarithromycin for one participant. None of the 31 eLung participants used the eDiary, as originally planned, despite the distribution of 42 eDiaries to 14 practices. Two of these 14 practices recruited three eLung participants, but none were provided with an eDiary. Nineteen eLung participants returned consent forms and were sent the paper EuroQol EQ-5D-3L and COPD Assessment Test questionnaires. Twelve eLung participants then completed the 4-week follow-up questionnaire. The remaining seven participants were sent 4-week questionnaires but did not return them.

Review of potential scenarios of fraud

A review was conducted of the risks for fraud in point-of-care trials. A list of potential scenarios for fraud and other GCP violations are outlined in Table 11 .

Sampling of general practitioner and patient participants

Based on previous research, a sample size of 15–20 GPs who declined and 20 GPs who agreed to participate in eLung was estimated as sufficient to reach data saturation. 92 If participant numbers allowed, purposive sampling aimed to include a broad range of GPs by geographical region, type of location (inner city, urban, rural, remote), size of practice, sex and recruitment level (i.e. a mix of high and low recruiters). A sample of 20 trial participants (10 in each arm of the trial) and 10 individuals who declined to participate in eLung was considered sufficient to achieve data saturation. 92 A purposive sampling method intended to aim for maximum variation to include both men and women in a range of locations. However, due to delays in initiating eLung, it was no longer possible to purposively sample participants for the QUEAN study.

Recruitment methods for general practitioner participants

Practices were initially approached about participation in the QUEAN study by either the CPRD organisation or by PCRN. CPRD’s data protection requirements are the reason for this, as direct contact between CPRD practices and an external organisation, such as the University of York, can occur only after consent by the practice. In the approach by CPRD, the QUEAN study recruitment was integrated with the recruitment process for Retropro and eLung. GPs in the 459 CPRD practices were sent two invitation letters from CPRD for the QUEAN study. The national PCRN co-ordinating centre and several regional offices also contacted practices in order to boost the QUEAN study recruitment (reaching practices that were not part of CPRD). GPs who were interested in receiving more information about the QUEAN study were asked to contact the University of York via e-mail, telephone or an expression of interest form. They were then provided with a full information pack and consent form. Following receipt of written consent, interviews were arranged with individual GP participants by the QUEAN study researcher by telephone or e-mail and confirmed in writing. Remuneration for GPs’ time to participate in the interview was provided at the standard PCRN rate.

Recruitment methods for patient participants

Given the sensitivities of inviting patients who declined to participate in eLung to participate in a related interview study, the initial approach regarding the QUEAN study was conducted by GPs. Following the invitation for eLung during the consultation using LEPIS, GPs could inform their patients about the QUEAN study providing them with a patient information sheet and an expression of interest form. Patients who were interested to receive more information could contact the University of York via telephone, e-mail or returning the completed expression of interest form. Following receipt of written consent, interviews were arranged with individual patient participants by the QUEAN study researcher by telephone or e-mail and confirmed in writing.

Ethics and research governance

The QUEAN study was approved by the South East Research Ethics Committee and over 100 R&D NHS trusts in England. The aim of this approach was to ensure approvals were in place in all NHS trusts in England prior to recruitment of the QUEAN study participants to avoid delay of interviews of newly recruited GP participants. In reality, the approval process continued for 18 months. The need for individual GP practices to obtain additional site-specific approvals instead of being listed as a patient identification centre resulted in further substantive delays. As a result, GPs who had declined to participate in eLung were not interviewed until a year or more after their decision and some patients were interviewed within 6 months of their invitation to participate in eLung. In Scotland, generic and local approval processes were completed for seven health boards and local practices within a month.

Data collection of general practitioner participants

In-depth interviews with GPs and patient participants used a semistructured topic guide (see Appendix 3 ). The questions were as open-ended as possible to enable participants to raise issues important to them. The wording and order of topics covered also varied considerably between interviews to ensure each interview was informal and open-ended to suit individual participants. 93 With the exception of one GP, all participants provided consent for interviews to be digitally recorded. The one GP telephoned the University of York and asked to be interviewed at that time, resulting in the interviewer taking notes while conducting the interview. Interviews with GPs were conducted over the telephone at a time of the GP’s convenience and lasted approximately 1 hour. Interviews with patient participants were conducted over the telephone or face to face at a location. Patient interviews lasted between 30 and 45 minutes.

Data analysis

Audio recordings were fully transcribed with anonymisation of all personal data. Interim analysis was carried out to develop the topic guide in the early stages of data collection to reflect emerging and important topics. 94 Framework method of analysis95 was adopted to organise the data. This approach was developed specifically for applied qualitative research in which the objectives of the study are often known in advance in agreement with a funding body. Although the framework approach starts from pre-set aims and objectives, as reflected in the QUEAN study topic guide, it adapts to reflect the original accounts and observations of the people studied based on a grounded and inductive approach. 96 The stages of framework analysis followed for the QUEAN study were those detailed by Pope et al. 96 The first stage was familiarisation, during which the researcher becomes immersed in the raw data by listening to the recordings and reading transcripts to list key ideas and recurrent themes. Those ideas and themes were then developed into a thematic framework which draws on the questions based on the study objectives and the novel views and experiences raised by the participants themselves. 96 The data in the transcripts were systematically indexed by annotating all the data with numerical codes from the thematic framework. This includes the constant comparison method to check and compare each item with the rest of the data to establish appropriate analytical categories. 96 This process is inclusive to ensure that categories are added to reflect as many as possible of the nuances or outlier views in the data. A charting process then organises the data according to the relevant part of the thematic framework to form charts. One chart per theme per participant was initially developed for theme by case analysis for the QUEAN study containing summaries of the participant views and experiences and references to verbatim text in the transcript. The individual charts were then grouped to include data entries for all participants in each subgroup. This dual process enabled comparative analysis to be carried out to allow data from individual participants and subgroups to be compared and contrasted. The final stage of framework analysis was mapping and interpretation to examine the associations between themes to provide explanations for the findings.

Positive attitudes of patients to research

It is interesting to note that, although the positive attitudes of patients to research was identified as a factor influencing the GP decision on participation in research (12 GPs), this was not identified as the most important factor by any GP. Descriptive data from the 12 GPs suggest that GPs are confident that their doctor–patient relationship will usually achieve a positive response from patients to the research if the GP takes the time to explain the study properly. This includes six GPs who provide care to populations experiencing high levels of deprivation. One exception to this view came from a GP based in a deprived area in the north who felt the patients are ‘too busy dealing with life stresses to be interested in research’. This GP had accepted participation in eLung, supporting the finding that this single factor is less likely to influence her decision to participate or not in a research study, but rather their ability to recruit among patients experiencing high levels of deprivation might be more difficult.

Time and capacity to implement a study

Nearly all GPs in this sample described a negatively changing context for research in primary care over recent decades due to significant increases in workload, reporting requirements and patient demand:

I don’t think we’ve ever worked so hard as at the moment.

GP 1012 (GP decliner)

The increasing workload and the extra QOF requirements have caused instant increases in our anti-depressant dosages. It’s grim.

GP 1009 (GP decliner)

The workload has quadrupled over the last 20 years, mainly due to the endless contract box ticking, non-contract box ticking, more administration and more patient demand.

GP 1019 (GP recruiting)

It’s all the QOF, all the tick boxes that you have to tick off . . . rather than deal with the individual, it is far more, a collection of diseases and you have to do x, y and z for each disease, it’s much more difficult to take the holistic view.

GP 1036 (incomplete set-up)

Within this context, the amount of time and ability a GP has to conduct a study has been identified as the most common and most important factor influencing the decision to participate in research. The amount of time a GP has may depend on the timing of when the GP is invited to participate in a research study. This is illustrated by a GP who had accepted participation in eLung but would have declined if asked at the point of completing the QUEAN interview because of the particular high level of clinical demand at that time. As indicated above, this suggests the timing of any recruitment activities to GPs should be considered within the context of their routine reporting or clinical commitments factoring in, for example, seasonal variations. The capacity of a GP to conduct a research study has been clearly identified by several GPs as determined by the simplicity of the study design and how easy it will be for the GP to incorporate it into their work:

I just don’t have enough time, even just an extra minute per consultation is not feasible at the moment.

GP 1036 (incomplete set-up)

If it doesn’t look too complicated and easy to understand and incorporate into daily business, it’s more attractive.

GP 1038 (withdrawal)

It depends how difficult the study is and how much it messes up the way we work.

GP 1029 (decliner)

Interest in research

General practitioners identified a personal interest in research as the second most important factor influencing their decision to participate in research. A strong level of personal interest in research appeared to be a predominant characteristic of 17 GPs who initially agreed to participate in eLung:

There’s so much bureaucracy and red tape with research it can put people off, some GPs are like, ‘Oh I’m not interested, I just can’ think about it because it’s just too much extra work. It’s got worse in the last 3 or 4 years, we’re getting used to QOF now although that keeps changing and there’s insurance reports, reports of the benefits agency, people having to appeal because their benefits have stopped so they need a report about their health for the last 20 years. There’s so much paperwork. Then the GP commissioning pulls them away from the practice for meetings, time is tight. You’ve got to be interested in research to overcome all that.

GP 1025 (recruited)

Nearly half of the GPs who identified an interest in research as an influencing (4/11) and important (4/7) factor identified a specific interest in EHR research and particularly EHR trials:

Personally, I want to be part of the national CPRD research.

GP 1005 (incomplete set-up)

I have a very strong interest theoretically in the method of running a trial within the CPRD database.

GP 1036 (incomplete set-up)

I have a personal interest in informing the study design in early stages.

GP 1030 (GP recruiting)

Conversely, three GP decliners who stated that they had no personal interest in research also had strong negative views towards research because of either a fear of litigation (one GP) or feeling extremely stressed and demotivated owing to workload (two GPs). The two GP decliners who expressed an interest in research were either disillusioned with the role of identifying participants for other studies or had a preference for local, practical research and a dislike of national academic-led research. It appears that a lack of strong personal interest in research in primary care enables the influence of other negative pressures or barriers to result in a decision to decline participation in a study. More positively, a strong personal interest in research appears to motivate a GP to accept participation in a study and potentially overcome those same barriers to research in primary care. This suggests that it is important to target recruitment for future trials to lead GPs for research or other GPs with a specific interest combined with broader educational and awareness strategies regarding trials within the database to generate interest in EHR research.

Benefit for local patients or population groups

The potential benefit of research to local patients or population groups was identified as an influential factor by approximately half of all GPs (n = 13), and as of primary importance by four GPs. Potential benefits to patients valued by GPs included improvements in the general standard of care received for a particular condition and an opportunity to take part in research which is often associated with improved health. Most GPs also identified the need for the research to be locally relevant and clinically important for the local population.

Adequate financial remuneration

Thirteen GPs also identified financial remuneration as an influencing factor. Five GPs identified the need for the study to generate a small profit including three GPs across different subgroups who considered this the most important factor influencing their decision to participate. This equates to 11% of GPs within the study sample (3/27) who identified making a profit as the most important factor influencing their decision. Of the eight GPs who identified the need for the study to be adequately remunerated to cover study costs, only one considered this the most important factor influencing his participation decision. The need to cover study costs is a consideration when a practice is involved in several studies, to cover staff costs or when motivation to conduct research is low because of the type of research or pressures of increasing workload:

Financial gain is helpful across some of your portfolio of studies as you would run into difficulty if they all ran at a loss.

GP 1031 (incomplete set-up)

Remuneration from [other organisations] has not been adequate, primarily because small numbers of patients were identified for the study. We don’t expect to make a profit but we’re not happy to run at a loss.

GP 1003 (decliner)

Previously I did research for nothing but it’s important to generate a small income now as it’s less motivating doing research identifying patients for [other organisations] rather than doing our own studies.

GP 1029 (decliner)

The remuneration needs to be adequate to cover the work it entails as we all have to think about staff costs these days and have we got the resources to fulfil the requirements.

GP 1025 (recruiter)

It’s important to make a profit for income as well as cover costs.

GP 1016 (withdrawal)

If the study’s not paid, it’s a definite no. Generous funding makes it more attractive.

GP 1037 (recruiter)

The remuneration should definitely cover costs and be enough to make a profit.

GP 1019 (recruiter)

One GP highlighted the importance of an appropriate payment to allow for the small number of patients who are likely to be recruited in some types of trial or practices.

Package of factors

As the relatively even spread of GP counts for each of the four influencing factors on the GP decision suggests, several factors appear to work in combination to result in an overall decision on whether or not the GP will participate in research generally. There appear to be various combinations of the factors identified as the most important, suggesting they are all important to some degree and need to be considered as a package to promote GP participation in future point-of-care trials:

Studies need to be locally relevant and practical, not academic and all too difficult and not worth the effort.

GP 1012 (decliner)

How feasible it is, how time consuming and awkward it looks and whether it’ll be worth the rewards.

GP 1019 (recruiter)

If the CPRD trials continue and provide the funds and make research interesting and not too labour intensive, then they might get some interest.

GP 1037 (recruiter)

CPRD offers a very attractive package, interesting studies for patient benefit and well renumerated.

GP 1038 (recruiter)

Notably, these factors were found to be similar for all GPs across groups of decliners, withdrawals, incomplete set-up and recruiters. This supports the earlier finding that a strong interest in research appears to be a key motivational factor within which the package of factors is considered as barriers or facilitators to research. The level of remuneration provided for Retropro and eLung was regarded as appropriate by the majority of GPs within each subgroup by eLung status.

Consistency with local prescribing guidance

Consistency of the eLung protocol with local prescribing guidance was the most frequently cited positive influence on the decision-making process for participation. In these cases, most GPs considered the study protocol to be consistent with their local prescribing guidelines on no antibiotics for a non-infective exacerbation of COPD. In two cases, the protocol provided a strong rationale to justify any contradiction to local prescribing guidelines. Several GPs identified the need to explain the rationale clearly to patients to overcome potential patient concerns and a small number considered the study would facilitate implementation of their ‘no antibiotic’ local prescribing guidelines. These views were expressed by GPs across all groups of decliner, withdrawal, incomplete set-up and recruiting. Two GPs expressed negative views that the protocol contradicted national and local guidelines on management of COPD exacerbations for patients who have previously been provided with antibiotics and corticosteroids for self-treatment at home. 97 The majority of GPs expressed a clear view that it is important for a trial protocol to be consistent with local prescribing guidelines.

Relevance of topic

The relevance of the study topic for eLung was the second most commonly cited positive influencing factor (n = 14) and the third most commonly cited negative factor (n = 4). Fourteen GPs, including GPs across all groups, to varying degrees, considered the eLung research topic as highly relevant and important clinically in the local population. One GP who subsequently withdrew because of problems with set-up considered the topic highly relevant to inform useful and needed advice to standardise practice in this area. Four GPs, all of whom were in the GP decliner group, did not find the topic relevant. The reason for this was the low incidence of COPD in their population (n = 3) or lack of interest in comparing one drug to doing nothing (n = 1). This GP did participate in Retropro because of a personal interest in comparing the side effects of different drugs, again highlighting the importance of personal interest as a key motivating factor in the decision-making process.

Feasibility to recruit

The feasibility to recruit patients into eLung was commonly cited as a positive (13 GPs) and a negative (10 GPs) influencing factor in the decision-making process for participation. Indeed, this was the most frequently cited negative factor influencing the GP decision on whether or not to take part in eLung with 6 of the 10 negative concerns being expressed by GP decliners. Specific concerns regarding feasibility to recruit included the impact of rescue packs on reducing patient numbers and potential confusion for patients, patient concerns regarding no antibiotics, lack of potentially eligible patients due to low incidence of COPD, and a lack of support in identifying and referring patients from other GPs:

We have a very low incidence of smoking and COPD so it’s not a priority issue and not worth the effort. It’s 20 years out of date as well, now we have rescue packs and micro-manage patients on the basis of need. It’s not very practical.

GP 1012

We had difficulty recruiting and following up in a previous trial so that put us off. The low number of people with COPD in the practice may have made it difficult to recruit.

GP 1003

It would have been difficult to recruit given so many patients would have been excluded having used their rescue packs or it would have contradicted local guidance and potentially go against good practice.

GP 1004

Patients without antibiotics are not likely to participate or comply. It would just cause disruption and argument.

GP 1009

In direct contrast, the positive factors underpinning the feasibility to recruit were based on the view that there will be a large number of eligible patients to recruit from within the local GP practice (nine GPs) and a minimal impact of rescue packs on participant numbers. The latter is based on the perceived views of GPs who have not started recruiting or completed set-up (three GPs) and a GP who has recruited four patients.

Computer-based pop-up alerts

The use of computer-based pop-up alerts to identify, screen and recruit patient participants was the fourth most influential factor both for GPs who considered this a positively influencing factor (n = 10) and for GPs who considered this a negatively influencing factor (n = 3). The negative views towards computer-based pop-up alerts appear to be based on a particularly strong dislike of this method from existing use in routine care and reporting. None of the GPs decliners who held these particularly strong negative views cited this as the main factor influencing their decision to decline participation in eLung. In contrast, the GPs’ positive views regarding pop-up alerts include excitement about their potential use in trials and, in particular, their time-saving attributes and efficiencies to reduce workload:

the alerts for QOF feel like brainwashing asking the same questions all the time.

GP 1009 (decliner)

the partners and GPs are not happy with additional pop ups, they’re clutter and an aggravation factor.

GP 1012 (decliner)

I’m not interested in a study with extra pop ups, there are too many already which interfere with the doctor-patient relationship.

GP 1034 (decliner)

I liked LEPIS to reduce the workload element of identifying patients through searches and getting a timely reminder removes the stress of having to remember the study all the time.

GP 1016 (withdrawal)

I thought of the trial as the ‘pop-up trial’, it was an interesting and memorable feature.

GP 1018 (incomplete set-up)

It’s useful to have an alert for an eligible patient.

GP 1017 (recruiter)

Hot recruitment method

Opportunistic recruitment of an eligible patient during the routine GP consultation was the most controversial element of eLung and generated the most frequently cited negative views, primarily from GP decliners (n = 6) but also from a small number of accepters (n = 3). The over-riding concern expressed by nine GPs regarding the concept of the hot recruitment method is a lack of time to include an additional task in the routine patient consultation and the subsequent lack of support from GPs to assist with recruitment:

Surgeries are already long and gruelling so only two GPs in the practice were willing to add an extra 5 minutes into their consultations for another study using hot recruitment.

GP 1029

Time is a fundamental problem.

GP 1032

Surgeries are always running late. Colleagues just won’t agree.

GP 1009

We’re a small busy practice and always running behind.

GP 1012

In contrast, GPs citing this as a positively influencing factor expressed their support for the concept in principle and a general ‘willingness to give it a go’. Only one of the six GPs expressing a positive view had used the hot recruitment method. The need for the recruitment method to be as simple and efficient as possible in practice was identified by GPs as a way of addressing their concerns and increasing motivation:

The hot method using pop-ups sounds like the best method. The truth is, for me, I prefer whatever is most efficient for a practice or as a group as a whole. Usually if I can do it there and then, that is going to be most efficient. If the study is then going to require 20–30 minutes at one time, I would say it’s going to be virtually impossible to do that. I get 10 minutes and I can allow 12–15 minutes so I’d much prefer to pass them onto someone else. If it’s going to take an extra 2–3 minutes on what I’ve already done, then I don’t see much point in passing it on to someone else.

GP 1004 (decliner)

Demands of study design on general practitioner time

Three of the GP decliners cited the study design for eLung as too demanding on GP time. These GPs had also expressed negative views regarding the hot recruitment method, pop-up alerts, patient concerns over no antibiotics, lack of local relevance of the topic, in addition to a lack of time to conduct the study. The same GPs also stated their lack of interest in research or eLung due to it not being locally practical and ‘academic’ or due to a feeling of being overwhelmed by the stress of daily clinical practice. None of these GPs had prior experience in research except one GP who had been involved in a small number of local projects. These three GPs provide care to population groups typically experiencing high levels of deprivation and from diverse ethnic minority backgrounds, including newly arrived immigrants. All three GPs are located in inner-city locations, two of which are in London. The following provides insight into the stresses of one of these busy, inner-city clinics and the interplay between several factors influencing GPs negative views in their decision-making process regarding participation in research:

I’m too set in my ways to change my antibiotic prescribing habits and I’m more than aware that a lot of the time that antibiotics are being prescribed for viral illnesses but because of expectations of patients or by habit from society that we’ve taught them to expect antibiotics for viral illnesses, I knew that one way or another, there’s a rod been made for own backs with prescribing and to break out of it would be extraordinarily difficult.

Just yesterday I was on triage which means no set appointments but all the incoming phone calls are put up on a screen, with a chesty cough etc. Some of that is antibiotic prescribing over the phone to patients you know but some cropped up where they’d contacted a colleague the day before with a child with ear-ache or something, and they’d been given advice but not given antibiotics and the next day the child is worse, well to actually bring that child in and check if they can just persevere and doesn’t need antibiotics, well, it would take time to give an explanation that we haven’t got and the next thing is they’d get them anyway through out-of-hours so, I’m well aware of the problems, if antibiotics had been a lot more unsafe to give, they wouldn’t have been overused in the way they are.

When we’re getting lots of our contacts with sore throats and minor upper respiratory tract infections or whatever, we can deal with those in a few moments, the patient’s happy, we’ve quite likely used an antibiotic NHS resource unnecessarily, the immediate problem is solved and to actually get through a triage session requires a lot of adrenalin, firing on all cylinders, flat out work for whichever of us is doing it whether its morning or afternoon and to actually take something that’s straight forward and make it more difficult it and lengthen it and make it harder would be intolerable! You’re running backwards for the first hour and a half as the calls come in faster than you can cope with but we’ve usually caught up by 12 o’clock.

GP 1034 (decliner)

In contrast, the four GPs within the ‘incomplete set-up’ group who considered the demands of the study design as a positive factor perceived the workload would be minimal using LEPIS alerts, EHRs to collect outcome data and the small numbers of patients. These GPs also expressed positive views on at least two other influencing factors, including the topic, use of computer-based alerts, the feasibility to recruit, hot recruitment or consistency with local prescribing plans. All four GPs were supportive of or had a strong interest in research, particularly studies which are within the EHR database or utilise a potentially quick and efficient method. Three of the GPs had previous experience in research. One GP described the particularly high levels of stress and workload in current primary care practice but had not experienced personal ill health associated as a result of that as had been the case with the two GP decliners who described similar demands. All four GPs provided primary care to predominantly white Caucasian population groups with some pockets of deprivation in mixed urban/rural locations.

These contrasting case studies further illustrate the emerging link between an underpinning interest in, and motivation towards, research in primary care with a highly positive and open attitude towards the novel aspects of a trial using computer-based alerts at the point of care. The need for additional support to encourage and support GPs working in particularly demanding practices to participate in research are discussed in Appendix 4 .

Summary of the factors influencing GP decision on participation in eLung

There are two sets of individual factors which influenced eLung participation. The first one was the decision to participate in any research study and the second one was the specifics of eLung as an example of a trial within the EHR database (see General practitioner theme 3: general practitioner views on hot recruitment using computer flagging and General practitioner theme 4: views of outcome data collection via electronic health records). These factors were consistent across subgroups of GPs who declined or accepted participation in eLung and/or Retropro. Two subsets of the most important factors influencing the decision to participate in any research project and eLung are illustrated in Figure 5 . For the majority of GPs, several influencing factors will work in combination within each of the two levels of the decision-making process. The findings (see Table 32 in Appendix 4 ) suggest that at least two and usually three generic factors on participation in any research project are viewed positively by each GP as a prerequisite before considering the more practical, project-specific factors (see Table 33 in Appendix 4 ). The inter-relationship of factors between the two sets was not explored explicitly within this study.

FIGURE 5.

Two subsets of most important factors influencing the GP decision to accept or decline participation in research. Outer circle: generic factors for any research project. Inner circle: project-specific factors.

The likelihood of a GP having a positive view towards several individual factors appears to be underpinned by a high level of personal interest and motivation towards research within the context of increasing pressures on GP time in primary care. Conversely, a complete lack of interest in, or negative view towards, research appears to be linked with a generally negative view towards a number of individual factors. This attitudinal factor appears to be a key driver influencing the decision of some GP decliners over and above the companion generic factors of ‘time/capacity’, ‘patient benefit’ or ‘financial remuneration’.

Views of hot recruitment

The lack of time to discuss the study and achieve informed consent within an unscheduled consultation was a serious concern expressed by nearly two-thirds of GPs (see Table 15 ). This included concerns about the ability to discuss the study properly, particularly with patients from non-English-speaking backgrounds or patients with concerns regarding anonymity of data and data confidentiality, need for the patient to have 24 hours to consider their decision to participate or not, and irritation at the impracticality of having to sign three paper-based consent forms. The acute nature of a COPD exacerbation raised concerns among a small number of GPs that the patient would be too sick to be able to give informed consent. Two GPs felt that it was more appropriate and potentially easier to achieve informed consent with a face-to-face discussion rather than a postal recruitment pack. The suggestions by GPs to adopt a more flexible approach to the recruitment method were primarily aimed at trying to overcome the concerns regarding informed consent. Other hypothetical concerns expressed by GPs regarding hot recruitment were likely difficulties recruiting patients who presented with several comorbidities, compounding the lack of time to combine recruitment with clinical care. One GP suggested a potential solution was to explain to the patient that the focus of the consultation that day would be limited to their COPD and to ask them to book another appointment to discuss any other concerns. Flexibility to opt out of recruiting a particular patient was identified as an important and positive function of LEPIS. This appeared to allay concerns of some GPs regarding the intrinsic difficulties of a lack of time.

Installation of Local Eligible Patient Identification Service software and its functionality

All 17 GPs who tried to have LEPIS installed onto their practice computer, including all three GPs who withdrew, seven of the eight GPs who failed to complete the set-up phase for eLung and all seven GPs who were recruiting patients into eLung, experienced significant problems with the installation, primarily due to local firewalls within the primary care trust or Clinical Commissioning Group. An inability to successfully install LEPIS was the reason for two GPs withdrawing from eLung and for seven of eight GPs failing to complete set-up and progress onto recruitment of patients. The delay to install LEPIS was the main reason why another GP withdrew from eLung, combined with its subsequent poor functionality and the lack of eligible patients owing to home use of antibiotics from ‘rescue packs’ in the last 2 weeks.

General practitioner recruiters (7/7) also experienced delays of 1–9 months for LEPIS installation. Three GPs found LEPIS worked satisfactorily following installation, two GPs had a mixed experience where it was working but incorrectly and two GPs did not consider it reliable enough to use for eLung. Problems regarding functionality included incorrect timing of the flag to alert the GP of an eligible patient (n = 4), no eligibility criteria (n = 1) and causing problems within the functionality of Vision software (n = 2). Examples of the GP feedback on the installation and functionality of LEPIS are:

At several points in it we were thinking, ‘Oh God, do we really want to carry on with this [LEPIS installation], it’s like wading through treacle’. When it was eventually working, the flag appeared to be triggered by the diagnosis which was after you’d given the patient a prescription and they’d left.

GP 1025

I had an awful bother with setting it [LEPIS] up.

GP 1038

LEPIS wasn’t working for a long time but it has improved, it’s still very glitchy but is now a useable tool. It only flags up on diagnosis for eLung and doesn’t go away. It was also removing functionality within VISION for repeat or acute prescriptions on the designated computer. LEPIS can be a tool which can deliver what you want it to but there’s still work to be done.

GP 1030

It would have been nice if LEPIS had been working reliably right at the start and we could just rely on opportunistic recruitment.

GP 1037

Experiences of hot recruitment

The actual experiences of GPs to recruit patients in the routine appointment were generally more positive than the hypothetical views of GPs. As shown in Table 16 , most GPs (3/4) reported the process took 5 minutes and was straightforward and feasible on most occasions. One GP found the process took 20 minutes and did not feel able to recruit when the surgery was running late. One GP experienced no problems in achieving informed consent. The GP who did find it difficult to gain informed consent within the time available suggested a preferred approach of presenting the study to patients at their annual COPD review. Examples of GP’s experiences of the consent process are:

The COPD population tend to be more deprived but no-one’s turned me down for it once I’ve explained it. This is why it is so important for the GP to believe in the ethos of the CPRD to be comfortable to tell patients about the study. I can put my hand on my heart and say I think it is a good idea to do the research.

GP 1038

Once you’ve got your head around the three consent forms and having to explain it all in the time, it’s not too bad. I have to select patients who will be interested and can understand in the time.

GP 1025

I have no concerns about the consenting process. We’re not talking about putting anyone on a horrendous drug which on-one knows about, we’re talking about everyday practice. Does the patient really need to go away and think about that when the doctor will put them on one treatment and another doctor will put them on another every day so why do we have an extra 24 hours to think about it for example?

GP 1030

Two GPs found the acute nature of the condition did limit the number of patients they felt able to recruit by half. In contrast, a similar number of GPs considered a treatment for an acute condition easier to administer in a hot recruitment situation than a change in a long-term drug. Three of the four GPs had used the study website instead of LEPIS to recruit owing to difficulties with the functionality of LEPIS. This was considered to work well and a useful back-up to LEPIS for two GPs. One GP experienced difficulties when the website would not accept the patient details. The GP’s experiences of hot recruitment in eLung included:

We created our own hot recruitment method by entering a code into the VISION pop-up for important messages so it comes up when we are consulting a patient. We then enter the data and randomise via the website which worked fine.

GP 1037

I was pleasantly surprised how easy it was when I actually did it. As a model of how these trials ought to be done, I think it’s quite good.

GP 1038

It’s impossible to do in the time available for both studies [eLung and Retropro] so if you’re very busy and running late in a surgery, you would probably ignore the recruitment opportunity. eLung takes 20 minutes. Once I got the hang of getting onto the website, it was very easy to use and randomisation is easy.

GP 1037

It’s not been easy, I started recruiting in summer so not many patients were presenting and the ones who do come in tend to need antibiotics or hospital. One eligible patient wanted antibiotics so didn’t consent. I have recruited one though and it was fine within the consultation, it just takes another five minutes to get the consent and randomise. Retropro was easier as there were more patients.

GP 1052

Some patients will be excluded due to mental illness or not being interested but I still would hope to recruit more typical, real world patients than the traditional clinical trials

GP 1019

Ease and efficiency of recruitment method to promote confidence

All four GPs who had experienced hot recruitment using computer-based pop-ups via LEPIS or the website expressed a clear view that, when the system is working, the recruitment process is fine and easy to do. A consistent and emerging theme from several GPs across all groups is a lack of confidence about whether or not it will work and the need for a simple and efficient process which will increase confidence to take the step of recruiting the first and most difficult patients:

I wasn’t sure how easy it would be but I thought I’d see how it went.

GP 1018

It’s all about the confidence to do the first one [recruit the first participant] and overcome the fear factor of ‘Am I going to be able to do it?’ Especially if you’re busy and the numbers are so small, is it worth the effort if you’re unsure? Retropro was bigger numbers and so more motivating in that way. I suspect some places [elung study sites] haven’t broken their duck [recruited] because they are not really sure what is going to happen.

GP 1038

Some studies are harder to start than others so we need to unpick what makes it hard. If the number of steps when you want to go [recruit] are complicated, then you may be less likely to start but if they’re done very slickly, then you are not . . . To start with, eLung failed on all these things as it was complicated. Even after you’ve done the training for eLung including the use of how to use the electronic diaries, I was already unsure what I was doing. If I was going to have to teach it to a patient, that meant I would need a substantial investment of my time to do it again, then my chances of remembering if I wasn’t going to do it for one or two months’ time, were close to zero. So we need to try and keep it simple, so what fell down here was, it’s a very common fault, everyone’s trying to come up with the perfect study but sometimes you make it so perfect to answer every question that comes along, it fails to achieve anything because it never gets off the ground.

The first or second times are the least likely times you are to recruit. What happens is when you first have a go to recruit, it either works well or is a total disaster and that determines how likely you are pre-test to recruit the next person. So where eLung has not been a success in my practice, because I’ve never recruited into eLung yet, therefore I’m thinking I don’t have a good familiarity, I’m aware the first time takes longer because I don’t know what I’m doing. I’ve also got all this in the background and so you never quite get started. So it’s self-fulfilling so that’s why it’s so important to actually get started in all these studies. The way that the study is set up, if it’s complicated or takes time to get set up or is very time consuming or you only see someone infrequently, then these are the reasons why you never get started and these studies often fail.

GP1030

I’d used it [LEPIS] in other studies so was confident it was doable.

GP 1031

The recruitment gets better as we get familiar with the study. So once we had done a few of them, you get to recruit ad hoc.

GP1052

To start with, it puts you off because you think ‘Oh God, I don’t know what to do and I don’t know how long it’s going to take’ but I think we’ve got over that now. Once you’ve enrolled a couple of patients you realise that it’s not going to take forever.

GP 1025

The delays in getting set up knock your confidence to start but when the system works, it’s actually very simple. It’s so important to get LEPIS working properly so it can be used more widely and reduce the workload involved in identifying and recruiting patients into trials.

GP 1037

Value of information analysis steps

The VOI analysis followed the following three steps, as proposed by Briggs et al. :113 (i) to construct a decision-analytic model to represent the decision problem; (ii) to characterise the current decision uncertainty by undertaking a probabilistic analysis of this model; and (iii) to establish the value of additional information. 113

Model structure

A decision-analytic model was developed for the VOI analysis in order to investigate the cost-effectiveness of prescribing antibiotics for COPD exacerbations in terms of the expected costs and effects for the two groups. The model, developed in Microsoft Excel (2010; Microsoft Corporation, Redmond, WA, USA), followed a decision tree approach and was based on the structure from Effing et al. ,124 as shown in Figure 6 . A cohort of 10,000 hypothetical COPD patients who present at their general practices with COPD exacerbations was followed in the model. The two alternatives considered by the analysis were ‘immediate antibiotics’ or ‘no immediate antibiotics’; there is a paucity of evidence regarding which is the more cost-effective option. We have taken the perspective of the UK NHS for the analysis125 (i.e. only direct costs to the NHS have been included). Owing to the 12-month time horizon of the model, we did not discount costs and health outcomes.

FIGURE 6.

Schematic of the decision tree model for management of COPD exacerbation.

The VOI analysis used data from literature and an observational cost-effectiveness analysis (CEA) which used data from CPRD (as described in Chapter 10 ). The probabilities used in the decision tree were derived from the observational CEA, as shown in Table 20 .

Quality-of-life outcomes

Health-related quality-of-life (HRQoL) data were derived from the literature. 126,127 The utility values used in the model are displayed in Table 21 and were applied similarly to both arms (i.e. to both the ‘antibiotics’ and the ‘no antibiotics’ arms). A sensitivity analysis was undertaken using utility values derived from the EuroQol EQ-5D-3L questionnaires completed by eLung participants.

Costs

Cost data were obtained from established national costing sources comprising NHS Reference Costs 2011/2012,128 the British National Formulary 129 and the Personal Social Services Research Unit Unit Costs of Health and Social Care 2012 130 as shown in Table 22 .

Incremental analysis

The results from the model were presented in terms of the incremental cost-effectiveness ratio (ICER). Specifically, the ICER was estimated in terms of the incremental cost per quality-adjusted life-years (QALY) (for cost–utility analysis). A cost-effectiveness acceptability curve (CEAC) was produced to explore the possibility that the use of antibiotics for COPD exacerbations will be cost-effective at different thresholds. 131 Net health benefit (NHB) was used to calculate the EVPI, where the NHB investigates if additional cost associated with a treatment is justified by the additional health gains. The NHB measure is the increase in effectiveness multiplied by the amount that the decision-maker is willing to pay per unit of increased effectiveness (i.e. the cost-effectiveness threshold), minus the increase in cost. The cost-effectiveness results were presented on a cost-effectiveness plane. 132

Expected value of perfect information calculations

The (individual patient) EVPI was calculated as the difference between the expected NHB given full information and the expected NHB given current information. The population EVPI was estimated by multiplying the individual per patient EVPI by the effective population, i.e. the annual population of patients who experience COPD exacerbations discounted over the lifetime of the treatment (assumed to be 10 years). The EVPI represents the maximum amount that a decision-maker should be willing to pay for further information to guide the adoption decision in the future; additional research should be considered if the EVPI exceeds the cost of research. The EVPPI was also calculated in order to provide more focus for further research, by identifying groups of inputs where it would be valuable to have more accurate estimates. Specifically, EVPPI estimates were generated for three sets of parameters: cost parameters, utility parameters and probabilities. The model used for the EVPPI was assumed to be linear for ease of computation.

Probabilistic analysis

In order to represent the decision uncertainty, a probabilistic sensitivity analysis was undertaken using 10,000 Monte Carlo simulations. Probabilistic analysis enables identification of the impact of different parameter values on the costs and effects of the different treatment options. Distributions were fitted to key parameters within the model; beta distributions (bound by zero and one) were used to capture the impact on uncertainty around probabilities and utilities. Cost parameters were modelled using the gamma distribution (producing non-negative values).

Cost-effectiveness results

The cost-effectiveness results from the decision tree model are presented in Table 23 . Patients who immediately received antibiotics experienced slightly more QALYs compared with patients who received no antibiotics. Each ‘antibiotics’ patient in the model was estimated to experience 0.730 QALYs [95% confidence interval (CI) 0.604 to 0.840 QALYs] for the 12-month time horizon, as opposed to 0.726 QALYs (95% CI 0.617 to 0.819 QALYs); an incremental effectiveness of 0.004 QALYs per patient. Hence, the QALYs gains were minimal.

The total cost per patient for COPD exacerbation management for the ‘antibiotics’ and ‘no antibiotics’ groups were estimated to be £329 (95% CI £291 to £371) and £448 (95% CI £398 to £501) respectively. Therefore, the use of antibiotics was associated with a cost saving of £119 per patient. Hence the analysis demonstrated that the use of antibiotics for COPD exacerbations was the dominant management strategy. As the QALYs were only marginally more favourable for the ‘antibiotics’ group, this finding should be interpreted with caution.

Figure 7 shows the cost-effectiveness plane for antibiotics compared with no antibiotics, obtained from the probabilistic sensitivity analysis. The sample estimates are spread entirely across the south-east and south-west quadrants, which demonstrates that antibiotic use was cost saving. The marginal QALY differences can be seen by the iteration results being concentrated close to zero for the incremental QALYs.

FIGURE 7.

Cost-effectiveness plane for antibiotics compared with no antibiotics.

The CEAC in Figure 8 illustrates the probability of antibiotic use and of no antibiotic use being cost-effective for different willingness-to-pay thresholds. For the NICE threshold of £20,000 per QALY, the probability of antibiotics being cost-effective is 0.56 compared with no antibiotics. For low thresholds, antibiotic use is likely to be cost-effective, with the curve for no antibiotic use showing the converse (i.e. that it is unlikely to be cost-effective for low thresholds).

FIGURE 8.

Cost-effectiveness acceptability curve.

Value of information results

The VOI analysis suggested that further research could be worthwhile ( Table 24 ). At a threshold of £20,000 per QALY, the individual total per patient EVPI was £567. The population EVPI was £3,979,601,769, which far exceeds the cost of a trial in this area (estimated to be £1.5M). Hence, this indicates that further research to inform the decision around use of antibiotics for COPD exacerbations could be worthwhile.

Specifying a minimum clinical difference in outcomes required to change clinical practice is one way to incorporate concerns that the results of the research may not be implemented perfectly (i.e. with 100% utilisation). For example, a larger clinical difference in effectiveness may need to be seen before the research has an impact on clinical practice. Requiring that further research must demonstrate larger differences in expected health benefits between antibiotics and no antibiotics reduces the value of research as larger differences are less likely to be found. This is illustrated in Figure 9 , which shows how the value of additional evidence falls as larger differences in NHBs between the treatments is needed to be demonstrated. When the minimum clinical difference between treatments is zero, the expected benefits of additional research is equal to the EVPI (i.e. all the uncertainty is resolved and it is expected that the results of research are implemented perfectly).

FIGURE 9.

Value of additional evidence and minimum clinical difference required.

The EVPPI explored the uncertainty of different parameters in the model. The uncertainty associated with the model derives from the utility (i.e. quality-of-life) values; the individual per patient EVPPI associated with utilities was £510. The VOI associated with cost parameters and probabilities was found to be negligible.

eLung quality-of-life data

We investigated the impact of using utility values estimated from eLung (based on the EQ-5D responses reported by the 12 participants who completed both the baseline and 4-week follow-up quality-of-life questionnaires). The mean utility score at baseline was lower for the ‘antibiotics’ group at 0.46 compared with 0.55 for the ‘no antibiotics’ group. At 4-weeks follow-up, the HRQoL had reduced for both groups: mean EQ-5D scores were 0.45 in the ‘antibiotics’ group compared with 0.52 in the ‘no antibiotics’ group. Using these scores, it was found that patients in the ‘antibiotics’ group experienced 0.075 fewer QALYs compared with those in the ‘no antibiotics’ group. Individual patient EVPI decreased to £39, population EVPI decreased to £274,525,916 and individual patient EVPPI for utilities fell to £5, for a threshold of £20,000 per QALY. Therefore, the results still indicate that further research in the area could be worthwhile.

Patient selection and inclusion

A cohort of patients was identified from CPRD by searching for patients who attended at their general practice for a COPD exacerbation between the year 2000 and April 2013. Individuals who were aged ≥ 40 years and had COPD exacerbation at least 1 year after the start of data collection were selected. In cases of multiple COPD exacerbations that occurred at 6 weeks apart, one was randomly selected. For records of COPD exacerbations within a 6-week period, the first one was used. The date of attendance for COPD exacerbation was defined as the index date. Using these data, two groups were formed based on whether or not antibiotics were prescribed during the index GP consultation. Patients who died before the cut-off point (i.e. at 4 weeks after the index visit) were excluded from the analysis.

Economic evaluation

We conducted a CEA to evaluate the use of antibiotics compared with no antibiotics for COPD exacerbation management. The analysis was undertaken from the perspective of the UK NHS and only direct health-care costs were included. In the base case, the time horizon of the evaluation was 4 weeks after the index visit for the COPD exacerbation. Owing to the short time horizon of the base-case analysis, discounting of costs or outcomes was not necessary. The following resource use areas were included in the analysis: GP visits, hospital admissions, community respiratory team referrals and all referrals. Antibiotics prescriptions and infections data were also assessed. Unit costs were applied based on established costing sources (see Table 22 ),128–130 with costs reported in 2012 figures.

Statistical analysis

Demographic and clinical variables were compared according to group. Poisson regression was used in the first instance to analyse the factors that determined the number of resource use counts over the analysis period. However, the data were found to significantly deviate from a Poisson distribution, hence negative binomial regression was used. Differences in resource use were expressed as rate ratios. For the mean difference in costs, a linear regression was used. We adjusted for covariates that could influence the results: age, sex, smoking history, body mass index and ethnicity.

Resource use

The resource utilisation data for the two groups are presented in Table 26 . For all outcomes, the antibiotics group used the least number of resources.

The rate ratio for GP consultations was 0.65 (95% CI 0.64 to 0.66) for the ‘antibiotics’ group compared with the ‘no antibiotics’ group ( Table 27 ). This indicates a significantly lower number of GP consultations in the ‘antibiotics’ group than in the ‘no antibiotics’ group. The ‘antibiotics’ group also had over three times fewer hospitalisations for the 4-week period (rate ratio 0.30, 95% CI 0.27 to 0.33). The infections analysis showed a slight decrease in the number of infections for the ‘antibiotics’ group, compared with the ‘no antibiotics’ group (rate ratio 0.91, 95% CI 0.84 to 0.99).

Costs

The mean adjusted cost per patient with a COPD exacerbation for the ‘antibiotics’ group was estimated to be £748, compared with £1911 for the ‘no antibiotics’ group ( Table 28 ).

The results indicate that the use of antibiotics for COPD exacerbations resulted in cost savings and an improvement in all outcomes analysed ( Table 29 ). Hence, the use of antibiotics was dominant over no antibiotics.

Step 1: development of an algorithm for electronic health record definition of end point of interest (in a non-trial population)

An algorithm for the EHR definition of the end point of interest will need to be developed, which could be challenging in EHR research databases given the heterogeneity in recording of data and varying levels of clinical details. Researchers have often used different definitions to extract data,134 ranging from very detailed case algorithms to broad definitions based on the presence of a single code. Researchers have also differed in which code lists should be used to extract data. 135 Rather than trying to classify each patient as a definite case or not, a better approach to dealing with heterogeneity in information may be to estimate the probability that a patient was correctly classified (i.e. positive predictive value). Several methods could be used to estimate the probability that the EHR algorithm for identifying cases was correct. It could be based on clinical review of possible cases providing the probability that cases as identified by the algorithm were confirmed in the review. A sample of cases could be reviewed by the clinicians caring for the relevant patients who are asked to confirm and validate the case outcomes and, ideally, provide supporting evidence independent of the EHR. Alternatively, the complete EHRs (from all linked data sets) could be extracted and reviewed by independent adjudicators. The positive predictive value for cases with, for example, heart attack that are consistently recorded in linked data sets (e.g. primary care records, hospital admissions and disease registry) would be much larger than that for cases recorded in a single source only. The availability of a linked database may be important for increasing the capture of outcomes and data quality. A recent English study that compared the recording in primary care, hospital admission and disease registry records found that each data source missed a substantial proportion (25–50%) of myocardial infarction events. It concluded that the failure to use linked EHRs may lead to biased estimates of the incidence and outcome of myocardial infarction. 136 This approach of the development of an algorithm for data extraction from EHRs has been used before, as was done, for example, in a study that looked at using EHRs for trial recruitment. 137

Step 2: validation of algorithm in a non-trial population

The validity of the algorithm used to identify the end points of interest in the EHRs could be evaluated by the analysis of incidence rates. As an example, the rate of heart attacks as estimated in the EHRs using the algorithm could be compared with estimates from literature. Such comparisons would also provide some evidence in the specificity of the case algorithm. An algorithm that would require extensive corroborating evidence in the EHRs would likely provide an underestimate of the true incidence rate as cases would be misclassified due to incomplete records. On the other hand, a non-specific algorithm could overestimate the incidence. Another analysis could consist of the evaluation of well-known associations, such as the Framingham risk score and CVD risk.

Step 3: identification of sites with unusual patterns of data quality in a non-trial population

Selection of sites with better quality of EHRs could be the third step. Sites with unusually low incidence of the end point of interest or with a pattern of unusual recording could be excluded from point-of-care trials. Figure 10 shows the results of cluster analysis conducted to identify practices with unusual patterns in data recording of CVD. The cluster analysis identified among the 295 CPRD practices, four groups of practices with statistically different patterns of data. The groups included 150 (50.8%), 52 (17.6%), 91 (30.8%) and 2 (0.7%) practices respectively. The two practices in the fourth group were found to have the lowest CVD incidence relative to the practices in the first group (adjusted relative rate 0.67, 95% CI 0.51 to 0.88).

FIGURE 10.

Incidence rates of CVD by calendar year in different groups of practices, stratified by cluster analysis. (a) Group 1; (b) group 2; (c) group 3; and (d) group 4. Each line represents the CVD incidence over time in a practice.

Step 4: application of algorithm to the trial population

The validated algorithm could then be applied to the trial population as long as the recording of data for trial patients would be similar to that outside the trial (i.e. the trial imposes no changes in data recording). In addition, the prospective, randomised, open, blinded, end point design could be used to review recorded end points of interest in a trial. 40 Adjudicators blinded to the randomised treatment would review all relevant information as extracted from the EHRs. In case of any questions or discrepancies between the various data sources, a case validation approach could be used. The EHR information could be summarised in a text report and then sent to the clinician at the site.

Lack of blinding to treatment and use of placebo

Point-of-care trials typically will have allocation concealment of the randomisation schedule (i.e. the person randomising the patient does not know what the next treatment allocation will be). However, patients and clinicians will typically not be blinded to the outcome of randomisation and know which treatment the patient was randomised to. Bias in unblinded trials may occur if perspectives vary between the treatments in the trial resulting in behavioural differences or the placebo effect. 138 The placebo effect has been described as a physiological effect caused by the meaning that the brain associates with the placebo. 139 An example of this is a study that reported substantive differences in symptoms of inflammatory bowel disease between an attentive patient–clinician relationship compared with being on the waiting list. 140 Bias may also occur if the measurements and recording of the trial outcomes vary between allocated treatments. Clinicians may refer patients selectively or outcome assessors may assess patients treated with the experimental intervention more favourably. 138 Blinding to treatment allocation is considered by many researchers to be a crucial method for reducing such observer bias. Tool for assessing risk of bias in randomised trials consider lack of blinding as a limitation. 141

Blinding would also not mimic routine care. These trials aim to randomise between different clinical decisions. In routine care, a clinician may choose between, for example, a statin or no treatment (placebo interventions are rarely used). Placebo effects and behavioural changes as a result of starting a treatment may be relevant for a clinical decision rather than being nuisance factors, as in an explanatory trial. As an example, a clinical decision to prescribe a statin should take into account any behavioural changes leading to a worsening diet, as this may impact on its effectiveness. Enck et al. 142 discussed how the placebo effect should be managed. They argued that the placebo effect should be minimised in explanatory trials in order to optimise drug–placebo differences, thus ensuring that the efficacy of the investigational drug can be truly evaluated. Once the drug is in clinical use, placebo effects should be maximised by harnessing patients’ expectations and learning mechanisms to improve treatment outcomes. 142

Measurement bias in the trial outcomes may also affect point-of-care trials. The degree of measurement bias will probably depend on the type of outcome. Diagnosis of death is unlikely to be influenced by awareness of trial medication. On the other hand, pain recording may be quite sensitive to lack of blinding. 138 A review of 146 meta-analyses found that effect estimates were exaggerated for subjective outcomes when there was inadequate or unclear allocation concealment or lack of blinding, whereas there was little evidence of bias in trials with objective outcomes. 143

The main end points of point-of-care trials could be major clinical outcomes, such as death or heart attacks. The prospective, randomised, open, blinded, end point design could then be implemented. 40 A point-of-care trial that obtains data from different data sets through linkage would further minimise bias due to differential diagnosis and recording of major clinical outcomes. A trial that uses data from various linked health-care data sets would not rely on the diagnosis, coding and recording of a single clinician. Careful consideration will need to be given to the choice of the outcome. The use of non-specific or composite outcomes could increase the proportion of events that are not causally related to the intervention, thus diluting the estimated effect towards the null. Conclusions made on the basis of the effect on a composite outcome may not be readily generalisable to other populations. 144 It is also important that the statistical analyses are prespecified and that the statistical analysis plan is published. EHRs may provide an opportunity for data dredging and selection of most favourable results. As reported previously, bias can be introduced if outcomes are not prespecified. 145

Patient-reported outcomes

The collection of patient-reported outcomes (such as quality of life) may be challenging in point-of-care trials. The possibility of measurement bias, because of the lack of blinding of patients, may affect the validity of more subjective end points, especially when patients have different perspectives on the randomised treatments (in eLung, many patients will have used antibiotics before). The feasibility of collecting these data during unscheduled consultations may be another challenge. As an example, the eDiaries were not used in eLung and the response rate for paper questionnaires was suboptimal. The lack of use of the eDiaries was likely related to the need to instruct trial participants about the device, which was not possible due to the lack of time during the unscheduled consultation. An alternative approach could be to utilise EHRs to develop surrogate measure for patient-reported outcomes. A study by Hutchings et al. 146 found that EHRs may have the capacity to develop these measures but that currently there was a wide under-reporting of symptoms in EHRs. 146 A future approach may be to collect patient-reported outcomes systematically during routine clinical care. For example, these data could be collected in the waiting rooms of clinics.

Assessment of trial eligibility

Screening for trial eligibility is the first step in the patient recruitment for point-of-care trials. One reviewer commented:

I tried to think about this in terms of a clinical question and trial that I would understand and settled on a comparison of sumatriptan versus non-steroidal anti-inflammatory drugs for patients with episodic migraine. Here’s where the problems begin. Diagnosis of migraine in primary care is notoriously inaccurate. The clinical question that would end up being answered is not sumatriptan versus non-steroidal anti-inflammatory drugs for migraine but rather sumatriptan versus non-steroidal anti-inflammatory drugs for patients GPs *think* have migraine.

A related challenge was experienced in Retropro: the calculation of CVD risk was found to be subjective and to vary substantially between different risk scores, without consistent guidance by NHS prescribing guidelines. 76 As a consequence, there were large differences for some patients between the assessments of CVD risk by the GP and principal investigator. There is no single answer to the ideal scope of eligibility criteria for point-of-care trials. One answer is that recruitment should be restricted to patients with a specific indication in which a biological effect has been observed in explanatory trials for the study intervention. An alternative answer is that point-of-care trials should be conducted if a substantial number of clinicians are already using the study intervention for a non-specific condition in order to address the uncertainty.

Heterogeneity of effects

Randomised trials typically address the question of whether or not an intervention is effective in an average patient. Average results can hide variations in individual responses. Analyses of subgroups of the trial population could evaluate any heterogeneity in effect, but they have limitations due to the possibility of random variation of effects across subgroups. Large trials that use EHRs could also provide an opportunity for data fishing expeditions, given the size of the population and the multitude of variables. It has been recommended that subgroups should be defined based on biological plausibility, or validated. 147 Heterogeneity of effects may be a large challenge in point-of-care trials compared with explanatory trials, given the deliberate intent to include broad patient populations.

Ioannidis and Lau148 proposed an interesting approach to systematically assessing heterogeneity which could be applied to point-of-care trials. A predictive statistical model would be built using known risk factors for trial outcomes. The variability in baseline risk can then be assessed and differences in risk evaluated. A homogeneous trial (i.e. with little variation in risk) may not yield results that are generalisable to more diverse population. On the other hand, substantive heterogeneity in baseline risk would indicate the need for evaluating the consistency of treatment effects at different levels of risk and subgroup analyses. 148 EHRs could also be used to compare the distribution of baseline risks in the population likely to be treated with the intervention with the distribution of risk among trial participants.

Generalisability of findings

Penston replied to our BMJ publication on the rationale for point-of-care trials14 with the following comment:

The authors make much of their contention that pragmatic trials will allow the data to be generalised. So much so that, in their conclusions, they imply that the external validity of the results will not be open to question. Given that there will be differences between doctors who participate in research and those who do not, that not all of the eligible patients will be recruited, and that the trial sample will not be drawn randomly from the underlying population, this is very hard to accept.

Sacristán et al. 149

Point-of-care trials should indeed be naturalistic and aim to enrol a ‘random’ sample of clinicians and patients. The original objective of this project was to have many clinicians each recruiting a few patients but we ended up with a few clinicians recruiting many patients. The critique by Preston is valid: point-of-care trials will provide generalisable findings only if clinicians and patients are a representative sample of the population of interest. EHR databases may provide an opportunity to measure characteristics of patients not recruited into a trial.

Lack of effect due to non-compliance

Bergmann supported the concept of point-of-care trials, but mentioned the following limitation:

The necessary intention to treat analysis may often conclude to an absence of difference between the two therapeutic strategies not because of an evidence-based clinical equivalence but because a weak evaluation of the efficacy or safety due to a too large variability linked to the erratic therapeutics compliance in usual conditions. This variability reduces the possibility to demonstrate the superiority of one of the tested treatments even if this superiority exists.

Sacristán et al. 149

Non-compliance is a nuisance in explanatory trials and needs to be minimised there. However, it is important to know for decision-making if a biologically effective medication is not being used by patients for whom it is appropriate. Erratic therapeutic compliance is therefore an end point of interest in pragmatic trials. Furthermore, the artificially high compliance achieved in explanatory trials by careful selection and regular instructions for patients may hide an improved performance in a normal practice setting for a less toxic intervention with fewer instructions.

Treatment contamination over time

Patients may change their treatment over time as highlighted by Bergmann:

After randomisation, the various doses, durations, stops, resumes, cross-over and individual modalities of the studied treatments as the different follows-up of the patients may trouble the demonstration of differences between the two compared treatments.

Sacristán et al. 149

Treatment contamination can indeed bias the study of biological effects as the exposures of interest are changed. However, it would be relevant to know that many patients switch from, for example, simvastatin to atorvastatin due to side effects. Treatment contamination is of direct interest in pragmatic trials as long as it reflects normal practice. Of course, it will be challenging to evaluate long-term side effects in pragmatic trials if there is treatment contamination. Statistical techniques, such as contamination-adjusted intention-to-treat analysis, may reduce bias due to treatment contamination. This method uses the statistical technique of instrumental variable analysis to address the contamination. 150

Concept of large trials

The concept of large trials has been described as deeply flawed by Penston. 151 His main reason for saying this is that the magnitude of treatment effects is often small in absolute rather than relative terms. Penston showed that less than 5% of patients in the CVD prevention trials derived any benefit from taking the medication for years (i.e. by not experiencing CVD). 151 Trials should be targeted at very high-risk people, thereby keeping the trials smaller. Clearly, the magnitude of absolute risk reductions with a treatment should be important for decision-making rather than only relative effects. However, low absolute event rates in trials are indicative not of flaws of randomisation but rather of imperfect identification of high-risk patients and targeting of treatment. Furthermore, patient preferences should be taken into account in deciding worthwhile levels of risk reduction.

Design of point-of-care trials

Retropro and eLung randomised individual patients to different treatments. An alternative design could have been the cluster trial. In cluster trials, entire areas or health service organisational units are randomly allocated to intervention or control groups with outcomes evaluated for individuals within each cluster. 152 Cluster randomised trials are increasingly used in public health and health services research. The main advantage of a cluster trial may be the easier implementation. For example, local prescribing guidelines currently restrict the choice of type of statin to simvastatin. Such policies could be amended to allow random allocation of each practice to the type of statin, which could be easily communicated to all practice GPs and monitored for adherence. However, there are several limitations with cluster trials. 153,154 One limitation occurs if there is variability between practices. For example, the use of antibiotics for COPD exacerbations substantially varies between practices, and this could reduce the statistical power of a cluster trial. Clinicians may also not allocate all of their patients to the intended treatment (e.g. withholding antibiotic treatment to patients with mild to moderate COPD exacerbations given their established clinical practice). However, cluster trials may be useful to compare the effects of different policies that are applied consistently (such as a policy of how to educate patients when to use their spare bottle of antibiotics).

To develop evidence and implement risk proportionality in trial governance and conduct

The experiences in this project and that of others46 do highlight the need for evidence of what works best and for risk proportionality in the governance and conduct of trials. GCP has been mostly based on expert opinion. It was also developed at a time when information was processed predominantly on paper. When dealing with the question of how to approach computer validation for EHRs from multiple systems and parties, we obtained advice that the best method for this was to print the EHRs and have the clinician sign the paper. GCP is also based on the notion that the records at the site are the source of trial information: the trial database needs to be consistent with the data at a site (i.e. source verification). However, this notion of a single data source is being replaced by a digital revolution with information coming from multiple places with different pieces of information. Loder et al. 170 have outlined that the ways we create, access and store information is changing rapidly. The challenge will be to develop the methods and approaches of how to extract reliable knowledge out of large amounts of data provided by digital technology. This challenge may only be met if the focus is on developing evidence rather than on following opinions and rules.

To develop strategies to increase recruitment in point-of-care trials

The nested qualitative process evaluation of eLung (the QUEAN study) found that recruitment of clinicians and patients into point-of-care trials was widely supported in principle and considered feasible for both acute and chronic conditions. Further research is needed to test effective strategies to improve the efficiency of point-of-care recruitment, particularly regarding innovative approaches to consent procedures. These challenges are fundamentally practical and potentially open to resolution. The greater challenge is the demanding context in which GPs operate and the underpinning requirement for a GP to have a high level of personal interest in research to address the many barriers to successfully implementing research among their daily clinical workload. Indeed, point-of-care trials provide an exciting opportunity to build on and broaden existing levels of GP interest in research despite limited time and capacity through the involvement in the full implementation of a trial with relatively minimal additional workload.

Based on the evidence of the QUEAN study and related literature, a package of recommendations is proposed to increase recruitment in point of care trials, one of the main barriers to the potential success of future point-of-care trials. The recommendations include strategies to address the specific technical and practical challenges unique to point-of-care recruitment, strategies to involve GPs and commissioners in identifying research priorities and informing design of successful implementation on the ground, combined with the crucial element of strategies to provide broader policy and infrastructural support for the promotion of simple and larger trials using EHRs to GPs ( Table 31 ). A comprehensive approach to both assessing the technical barriers and enabling the promotion and support of successful recruitment and implementation is more likely to improve recruitment at each stage of the process. This is important for point-of-care trials which require wide inclusion of GPs to increase external generalisability and high retention rates to maintain internal validity. 89 It also recognises the influence of several generic and project-specific factors on the recruitment decision for a research study as identified by the QUEAN study. Although this would require investment in the short term, including site-specific support to enable participating GPs to recruit successfully and positively, this would be expected to reduce over time as confidence and familiarity with the trial method increases as an integral part of a learning health-care system.

To increase the involvement of patients and clinicians in point-of-care trials

In addition to the formal evaluation of the factors influencing recruitment into point-of-care trials in the QUEAN study, the two patient representatives on the Trial Steering Committee each provided their views on how to progress with point-of-care trials (see Appendix 5 ). The first necessity is to build a clear sense of common purpose and urgency about facilitating the choice of research questions. Second, expediting the set-up process for point-of-care research, particularly with regard to the burden of obtaining research governance approvals, is also required. The third is to establish a team committed to leadership, capable of sustaining momentum and resourced to do so. Early objectives must be to demonstrate the effectiveness and benefits of point-of-care randomised trial research in resolving high profile unresolved uncertainty dilemmas of common concern to clinicians and patients. Pioneer projects should affect both primary and secondary care and ideally point to cost savings to demonstrate that, over the medium term, such research can be financially self-sustaining, as well as providing evidence resulting in patient benefit. Furthermore, alternative models for engaging patients and seeking their consent should be explored, including waiting room posters or rolling TV advertisements in the waiting room of general practices.

The two lead GP investigators also provided their views on how to increase GP involvement in point-of-care trials (see Appendix 5 ). The challenge in point-of-care trials is to simplify the initiation and conduct and to have research questions that will be of concrete interest to clinicians. Such simplicity should make it easier to recruit clinicians and patients. In the UK, the PCRN’s Research Sites Initiative should be utilised to provide support for local feasibility and to include successful participation in point-of-care studies (and maintaining the software tools) as part of the criteria for participation in research site initiatives schemes. Local Clinical Research networks could be used to engage with Clinical Commissioning Groups, NHS and Commissioning Support Units (who maintain practice IT) to ensure wider and more responsive support for the technical challenges in these studies. One of our qualitative studies found that seeking consent for trial participation was a barrier for GPs but not for nurses. The role of nurses in point-of-care trials should be further explored.

To obtain patient views on how to deal with and communicate uncertainty

It is currently difficult to initiate trials even for routinely used interventions, as we and others have experienced. The trial regulation and governance system are complex and only few clinicians participate. The consequence of this complexity may be that trials are not routinely conducted and uncertainty is not addressed. There is currently a paradox in how the health-care system deals with uncertainty. Richard Smithells noted that he needed permission to give a treatment to half of his patients (to find out whether it did more good than harm), but that he did not need any permission if he decided to give the treatment to all of his patients. 45 Informed consent requirements also are very different with trials requiring lengthy consent forms in contrast to brief discussions in usual care. 171 COPD patients are not informed routinely that their exacerbation would be treated differently by another clinician,26 while many obstacles are put into the way for anyone wanting to initiate a trial in order to resolve this variability. The fundamental question is how patients do view this apparent double standard in the handling and disclosure of uncertainty. A survey of over 1000 adults found that 97% of respondents agreed that the NHS has a duty to determine the safety and effectiveness of the drugs its doctors prescribe. 172

To develop consent procedures informed by patient views

The informed consent procedures should be designed in collaboration with patients and be based on empirical evidence on what kind of process best informs participants. However, the GCP standard, which has come to be viewed as canonical for trial governance, was based on expert opinion and has little basis in empirical evidence. A recent systematic review found that there is limited empirical evidence of what information potential participants want to know about research when they are making the decision to take part. 173 There is a need to understand better the type and amount of information people want in order to support their individual decision-making. It would also be very useful if the consent procedures could take place some time prior to randomisation, especially for hot recruitment of acutely ill patients. COPD patients could have been approached in eLung and asked for consent to randomisation for a future exacerbation rather than at the time when they were acutely ill. Rather than lengthy paper forms with a paternalistic content determined by experts, consent forms should be tailored to individual needs and use modern technology to communicate.

To measure and acknowledge systematically uncertainty in guidelines

It is currently no longer possible to initiate Retropro or eLung. Although there is scientific equipoise in the comparative effectiveness of different types of statins and in the effects of antibiotics in mild and moderate COPD exacerbations, there is no equipoise for clinicians due to recommendation in guidelines. For statins, the proposed 2014 NICE guidelines for primary prevention of CVD now recommend atorvastatin over other types of statins. This is despite the finding that there was no clinical and statistical difference in their meta-analysis of trials between different types of statins (http://guidance.nice.org.uk/CG/WaveR/123). 174 This recommendation appears to have been based on a cost-effectiveness model which ignored this clinical and statistical uncertainty. It is important that uncertainty is systematically measured and acknowledged in guidelines. Otherwise, we will remain in a vicious circle in which a guideline is based on uncertain evidence but research cannot be done because of the guideline.

To develop cost–benefit analysis to assess excess treatment costs

The excess treatment costs were found to be a considerable barrier to point-of-care trials. The question is who should bear the costs of the routine treatments provided for a trial. The NHS would be the main financial beneficiary for a full Retropro trial with clinically significant findings. If atorvastatin was 10% more effective than simvastatin in reducing CVD, the NHS would save the costs of over 2400 heart attacks for all patients starting statins in a single year (as outlined in Chapter 1 ). This saving would clearly offset the excess treatment costs of a full Retropro trial. The handling of excess treatment costs should be based on a comparison of the future cost benefits of likely trial findings with the excess treatment costs of a trial.

To develop statistical models for the measurement of electronic health record data quality

There are currently over 300 EHR research databases in 45 countries with different characteristics. 175 Data quality is of course very important, and not all clinical outcomes can be measured accurately solely from the data recorded in the EHRs. Information in EHR research databases can also change substantially over time. There is a clear research need to develop and adopt systematic, statistically based methods of data quality assessment. 176 Furthermore, the development of common data models and dictionaries could simplify research across different EHR research databases.

To test risk prediction and patient identification strategies in randomised trials

There were large differences for some patients between the assessments of CVD risk by the GP and principal investigator. We initiated research that explored the variability between practices in statin prescribing between practices76 and in the performance of the Framingham, ASSIGN and QRISK2 risk predictions. We found that statin prescribing varied substantially between practices76 and that the existing risk scores did not predict well CVD risk in individual patients. Point-of-care trials will be useful only if the right patients are identified consistently. Therefore, risk prediction strategies should be subjected to randomised trials.

To further experiment with point-of-care trials

Michael Lauer of the National Institutes of Health has recently expressed a willingness to experiment, during a meeting at the US Institute of Medicine about simple trials. Emerging technologies should be embedded into existing projects and be allowed to fail often but inexpensively in smaller experiments. This allows for a stepwise approach to learning what works as long as lessons are captured and shared. 177 Both EHRs and pragmatic trials may be considered emerging technologies. Although the concept of pragmatic trials was described over 40 years ago,1 they are conducted less frequently. 178 The combination of randomisation and simple data collection through EHRs can provide the opportunity for a stepwise change in quality improvement in the health-care system and wider use of point-of-care trials. Researchers will need to share data and knowledge openly with a view to improving health outcomes for all. 170

Reith et al. 179 concluded in their recent review of barriers to trials:

It is becoming increasingly clear that more extensive use of health records and informatics platforms, along with more refined ethical approaches characterised by the expectation that participation in clinical trials is the norm rather than the exception, could support a dramatic increase in the emergence of definite evidence about treatments. 179

Reith C, Landray M, Devereaux PJ, Bosch J, Granger CB, Baigent C, et al. Randomized clinical trials – removing unnecessary obstacles. N Engl J Med 2013;369:1061–5. Quote reproduced with permission

We conclude by quoting the comments by Sacristán et al. :149

The combination of experiments and observations in daily practice requires important regulatory and cultural changes oriented to eliminate the barriers between clinical practice and clinical research, realising that all research and all clinical actions begin at the patient’s bedside and that every medical act is structured like an experiment. The increasing use of EHR should contribute to the disappearance of walls between doctors who carry out research, and doctors who do not, between patients who participate in RCTs and the ‘real’ patients that doctors see every day, between the clinical research form used in RCTs and the electronic medical history. The systematic use of EHR to conduct RCTs could contribute to completing the circle that began with evidence-based medicine with the new concept of medicine-based evidence. The real challenge is not the technical infrastructure to implement randomised database studies, but understanding that, in the context of a patient-centred medicine, clinical research and clinical practice are the two faces of the same coin. 149

Sacristán JA. Rapid Response BMJ Re: Pragmatic randomised trials using routine electronic health records: putting them to the test, 2012; URL: www.bmj.com/content/344/bmj.e55?tab=responses (accessed 2 July 2012). Quote reproduced with permission

Roles and responsibilities

The role of the Steering Committee is to provide oversight of the conduct of the two trials. This includes oversight of the practical aspects of the study as well as ensuring that the study continues to be run in a way which is both safe for the patients and provides appropriate safety and efficacy data to the sponsor and investigators. In discharging its safety role, the Steering Committee will work in conjunction with the Data Monitoring Board that will also be established for the trials.

Specific responsibilities of the Steering Committee include, but are not limited to, the following:

• to provide overall supervision of the trials

• to take steps to reduce deviations from the protocol to a minimum

• periodic review of the progress of the study

• to review safety data; this review is typically done blinded to treatment allocation (in case of a major safety concern, the Steering Committee can request unblinding and the review then can be done in an unblinded manner)

• to resolve any differences within the research team or between research team and sponsor (London School of Hygiene & Tropical Medicine) on the data management and monitoring procedures in the trials or any recommendations for modifications to the protocol.

The Steering Committee will have ultimate responsibility for the trials and will assume primacy over the Data Monitoring Committee or principal investigator. The Steering Committee can prematurely terminate the trials. The sponsor and principal investigator will agree, in writing prior to the start of the study, to the charter of the Steering Committee.

Membership

The Steering Committee will consist of at least four members, including three members who are independent of the MHRA. A patient representative will be invited. The sponsor and the funder of eLung will need to approve the nomination of the Steering Committee. Membership consists of persons who have no financial, scientific, or other competing interests with the trials. Three members (including the Chairman or designee) will constitute a quorum. The principal investigator and a representative of the financial funder of the trials may attend the meetings, but will have no voting status. The Steering Committee may request that the principal investigator and the funder representative do not attend all or part of any meeting. Members will be reimbursed for any reasonable travel, accommodation or other costs (e.g. telephone) incurred.

All potential members will have sight of the protocol and be invited to comment before agreeing to join the Steering Committee. If a potential member has major reservations about the trials they should report these to the principal investigator and sponsor and may decide not to accept the invitation to join. Steering Committee members should be independent and constructively critical of the ongoing trials, but also supportive of aims and methods of the trial.

There will be no formal contract between Steering Committee members and the sponsor.

Committee process

The Steering Committee will meet prior to the start of the study and 6–12 months after the start of the study. The Steering Committee may also meet on an ad hoc basis should the need arise. Meetings may take place by teleconference, by videoconference, or face to face, whichever is agreed to be most appropriate. Urgent issues may be communicated by e-mail by the principal investigator to the Steering Committee, where appropriate. Members will be reimbursed for any reasonable travel, accommodation or other costs (e.g. telephone) incurred. There will be no formal legal contract between members and the sponsor.

Minutes

A set of minutes (drafted by a member of the research team) will be produced for each meeting of the Steering Committee. These will be circulated in draft form to all members within 1 week of the meeting date. Members then have up to 2 weeks in which to provide comments or amendments, after which the minutes will be considered to be final. The Chair (or in his/her absence from the meeting a nominated deputy) will sign a copy of the final minutes. A copy of the final minutes will be filed in the trial master file.

Confidentiality

All materials, discussions and proceedings of the Steering Committee are completely confidential. Members and other participants of the Steering Committee meetings are expected to maintain confidentiality.

Roles and responsibilities

The role of the Data Monitoring Committee is to safeguard the interests of the trial’s participants and to monitor the data collected in the trials. Specific responsibilities of the Data Monitoring Committee include, but are not limited to, the following:

• to agree to and evaluate the data management and monitoring procedures of the trials as proposed by the research team

• to assess recruitment figures and data quality, including completeness

• to assess the extent of protocol deviations (including a comparison of patients enrolled in the trials and other patients in the research database using the same medication)

• to review safety data, including line-listings of case reports of suspected ADRs and of iatrogenic conditions and to request further analyses; this review will be done blinded to treatment allocation; the Data Monitoring Committee will inform the Steering Committee of any major safety concerns and request their unblinded review of safety data

• to review the results of the fraud detection procedures

• to implement early stopping rules for the trials (e.g. Haybittle–Peto rule).

The Data Monitoring Committee can recommend modifications to the data management and monitoring procedures in the trials or to the protocol. Every effort will be made to reach a consensus within the Data Monitoring Committee and with the principal investigator. In case of disagreement on recommended modifications, the Steering Committee will decide. The sponsor and principal investigator will agree, in writing prior to the start of the study, to the charter of the Data Monitoring Committee.

Membership

The Data Monitoring Committee will consist of three members. Two members will constitute a quorum. In the two trials the members of the Data Monitoring Committee will consist of co-investigators (rather than independent members as will be the case for later studies). The reason for this is that these are feasibility trials lacking statistical power to detect effects on major clinical outcomes; the objective of these studies will be to develop the overall structure and procedures for trials.

Committee process

The Data Monitoring Committee will meet prior to the start of the study and 6–12 months after the start of the study. The Data Monitoring Committee may also meet on an ad hoc basis should the need arise. Meetings may take place by teleconference, by videoconference, or face to face, whichever is agreed to be most appropriate. Urgent issues may be communicated by e-mail. Members will be reimbursed for any reasonable travel, accommodation or other costs (e.g. telephone) incurred. There will be no formal legal contract between members and the sponsor.

Minutes

A set of minutes will be produced for each meeting of the Data Monitoring Committee. These will be circulated in draft form to all members within 1 week of the meeting date. Members then have up to 2 weeks in which to provide comments or amendments, after which the minutes will be considered to be final. The Chair (or in his/her absence from the meeting a nominated deputy) will sign a copy of the final minutes. A copy of the final minutes will be filed in the trial master file.

Confidentiality

All materials, discussions and proceedings of the Data Monitoring Committee are completely confidential. Members and other participants of the Data Monitoring Committee meetings are expected to maintain confidentiality.

All interviewees (agreed or declined eLung pilot trial)

Can you tell us what prompted you to visit the GP that day?

Prompt – brief history of condition and history of exacerbations, how are exacerbations normally treated by GP.

Did you find the GP visit helpful?

Prompt – good and bad aspects of consultation.

Are you feeling better?

Prompt – further visits, treatments, care pathway.

Can you tell me what you can remember about being asked to take part in the trial?

Prompt – perhaps have patient information sheet, who mentioned the trial to you in the first place, who provided the information.

What are your views on why this research is being carried out?

Prompt – ascertain information about how understand medical research evidence and why trial was needed in this particular instance.

eLung pilot trial participants

What were your reasons for agreeing to take part in the COPD trial?

Prompt – hope of getting a better treatment, hope of improving treatments in the future for other patients, relationship with the GP.

What was your experience of being asked to take part in the study?

Prompt – views on how you were approached during the consultation, what information was given, time to consider your decision.

Did you feel well enough to think about your decision and give informed consent about taking part in the study?

Prompt – feel the need to discuss with family or friends, able to understand and consider the information.

Were you happy with the treatment you were given after agreeing to be in the study?

Prompt – did understand they allocation process, did they understand equipoise, how did the allocation make them feel about the trial process.

Did you use the prescription and take the treatment?

Prompt – GP prescription or rescue pack, antibiotics and/or steroids, completed the course?

Are you happy with data being taken from your patient records by a third party in London to find out how you responded to the treatment?

Prompt – anonymised, links to hospital data, prefer to fill in a questionnaire, on one or several occasions for years to come.

What has been your experience of taking part in this study?

Prompt –positive/negative factors and why.

Is there anything you would have changed about the study to make it easier for you?

Do you have any advice for researchers about getting people involved in this type of research?

eLung pilot trial decliners

Can you explain what were your reasons for not wanting to take part in the COPD trial?

Prompt – wanted antibiotic treatment, not interested in this trial, not well enough to think about it, not enough information or time to think about it, not clear what was involved, the relationship with the practitioner, did not believe it would make a difference.

Could the research team have done anything differently which might have made you want to take part in the trial?

What are your thoughts on what researchers could do to get more people involved in this type of research?

Study documentation and communication with general practitioners

Most GPs considered the eLung documentation for GPs and patients to be clear, particularly regarding core information on the aims of the pilot trial, inclusion/exclusion criteria of eligible patients and study outcomes. Nearly all GPs were aware the trial was a pilot with the primary aim of exploring a methodological question. Some aspects of the study design were less clear for a handful of GPs including three GPs who incorrectly believed their practice had been allocated as a control site and one GP who was also not clear whether the invitation to participate was targeted to a single or multiple GP(s) within the practice. Many GPs were not clear or confident about how the hot recruitment would work in practice and identified the need for more personalised and/or practical input in the early stages of recruitment. A small number of GPs expressed concerns regarding resending of entire documents by e-mail following specific changes to a document resulting in the GP having to reread entire documents several times. Other concerns included the need for study documents to address data protection requirements for remote collection and analysis of patient data and provide clear information on these issues in the patient information sheet. The main concern expressed by several GPs across all groups was regarding access to timely and appropriate support throughout the set-up phase, particularly during the widespread difficulties experienced with the installation of LEPIS. Site visits by the research team to resolve the technical difficulties were highly valued by all GPs. GP recommendations to improve information and communication strategies for future trials include the following:

1. Avoid GP recruitment during busy periods such as reporting for QOF or winter flu epidemics.

2. E-mail communication with GPs should be used sparingly and new information should be separated out from previous versions of documents.

3. Clear and simple information to GPs including a one-page summary of the study design and exclusion criteria.

4. Clear and simple information for patients, particularly on data protection for remote access of patient data and a single signature for consent. Patients who have signed a disclaimer should be excluded.

5. Study materials should include an online demonstration, via a web-based podcast, for example, to demonstrate how hot recruitment using pop-ups actually works to screen, recruit and randomise patients in different scenarios. This would boost confidence for the lead GP and could be used as a training aid for other GPs.

6. Each practice should receive a telephone call from the research team to explain the entire study alongside the invitation letter as part of the initial recruitment process.

7. The option of a face-to-face visit for GPs who have agreed to take part to explain the study in more detail or address specific concerns would boost confidence and motivation for GPs who are new to this trial method.

   1. – A face-to-face meeting would also provide the opportunity to promote the study design to multiple GPs to increase support to the lead GP for improved patient recruitment and increase awareness and interest in trials of this method among a broader base of GPs.

Input from general practitioners to inform future trials

Nearly all GPs within all subgroups of eLung status believed it would be extremely useful for practising GPs to have input into the design of trials and how they would work on the ground to reflect the realities and variation of how general practice works, and to identify priority topics of local relevance and related outcome measures for future trials. Most GPs suggested that the lead GP on research would be the most appropriate person to provide input on the practicalities of the study design, with a small number of GPs identifying the potential need for financial remuneration to backfill GP time if this task took the GP out of clinical practice. Strategies to identify an ‘Advisory Group of Lead GPs’ with an interest in this role included contact by the EHR database to all lead GPs on research to invite expressions of interest (via internet-based questionnaires for example) and/or use of informal networks with GPs known by the EHR database to have an interest in this role. A small number of GPs did not consider GPs to be the appropriate choice to identify priority topics of research interest for future trials. Instead, Clinical Commissioning Groups and Local Medical Committees were suggested as having the relevant knowledge and expertise to fulfil this role. GP recommendations on input to improve the design of future trials are as follows:

1. An ‘Advisory Group’ of Lead GPs on research should be identified by the CPRD to provide input on the practical design of a study and how it is implemented on the ground in different practice settings.

2. Lead GPs, Clinical Commissioning Groups and Local Medical Committees could assist with identifying priority research topics and key outcome measures with local clinical relevance.

3. Financial remuneration may be necessary to backfill GP time from clinical practice.

Flexible strategies to recruit patients

The need for flexible recruitment strategies to combine more traditional cold recruitment strategies with hot recruitment within the consultation was recommended by several GPs across all groups. These GP recommendations were primarily aiming to ensure sufficient time was available to achieve informed consent while also removing any elements of the recruitment process from the consultation to increase efficiencies and feasibility of the hot recruitment process. GP recommendations on flexible strategies to improve recruitment of patients are as follows:

1. Flexibility for each GP practice to implement a range of recruitment strategies which support hot recruitment using computer-based alerts where appropriate or a mix of hot and cold recruitment as necessary. Specific strategies include:

   1. – introducing patients to study materials at routine annual checks/clinics

   2. – screening patients at triage, or appointment booking and allocate a 20 minute appointment

   3. – text messaging to patients to check eligibility and for appointment reminders

   4. – postal letter to eligible patients to alert them to study and recommend change of usual care (e.g. contacting surgery prior to using rescue pack), if necessary

   5. – option for GP to connect patients to a CPRD research nurse via the internet to discuss the patient information and consenting process prior to the GP completing consent and recruitment

   6. – option for GP to enter some of the patient data onto the system at the end of surgery, allowing more time in the consultation for consenting

   7. – important to keep the flexibility for a GP to opt out of recruiting a particular patient if the GP is too busy or patient appears too unwell, for example.

Computer-based alert system

The over-riding concern regarding the computer-based alert system for GPs across all subgroups was the need for it to enhance the efficiency and simplicity of the trial to encourage GPs to be willing to take this method on board and take part in the trial and/or to increase confidence in its feasibility. The study website was seen as a valuable alternative to the LEPIS alert system. GP recommendations on how to increase support for use of computer-based alert systems, such as LEPIS, are as follows:

1. Outstanding problems regarding installation and functionality of LEPIS need to be resolved prior to use in a full trial.

2. Screen-based pop-ups must be extremely quick, easy to use and based on data which is routinely collected during the consultation.

3. Data collected in response to the study alerts must be linked directly to the patient record to avoid duplication of data entry.

4. Option for GPs to recruit via the study website as an alternative to LEPIS.

5. Study data should be entered as free text or standard codes.

6. CPRD would interpret study data via remote download with assistance from GPs as required on individual records.

Additional support for vulnerable practices

Over two-thirds of GPs in the QUEAN study, including those who had declined, accepted or recruited participation in eLung, are located in areas with high levels of deprivation and are from a variety of ethnic backgrounds. All three GP participants who are based in London had declined participation in eLung. It appears that the additional factors of a highly mobile population which is generally not compliant with GP advice, due to a suspicion of authorities in the case of immigrant populations or a consumer-led attitude of moving between services until patient needs are met, may be contributing to the decision-making process on participation in research generally or eLung. GPs employed in small practices can also experience additional difficulties, for example increased variation in quality of data entry and coding due to a greater reliance on locums to fulfil staffing quotas. GP recommendations on strategies to provide additional support for vulnerable practices are as follows:

1. Recruitment of a collaboration of service providers across a London region who would provide consistent advice and practice to patients regardless of mobility and patient demand.

2. Financial incentives for patients to participate in the study.

3. Option of additional administrative support to aid, for example, complementary cold recruitment strategies, checking data quality.

Incentives for general practitioners

The GP recommendations of incentives for GPs including financial, educational and developmental ones are as follows:

1. Promotion of the novel trial method to utilise EHRs for collection of outcome data is likely to generate an underpinning personal interest and motivation for most GPs to encourage participation in future trials.

2. Adequate per participant remuneration, consistent with levels provided for eLung to cover costs and potentially generate a small profit.

3. Formulae to calculate flexible financial remuneration packages to cover costs for combined hot and cold recruitment strategies.

4. Packaging of remuneration of other ‘enhanced services’ like QOF targets which improve income streams.

5. GP training on research topic.

6. Quick and timely feedback to practices on recruitment rates including acknowledgement of high recruiters, for example a certificate to display in the practice to acknowledge valued input of patients.

7. Research can be promoted as an opportunity for revalidation within the Good Medical Practice Framework.

Study findings suggest these recommendations should be considered in their entirety to identify a mix of strategies which will have a bearing on the range of generic and project-specific factors identified as important in influencing the GP decision to participate in future point-of-care trials.

Large simple trials in primary care also provide an opportunity for increased personal enjoyment and satisfaction for GPs who have limited time but enjoy research and being involved in all aspects of implementing a trial. Broad-based promotion of the efficiencies of hot recruitment, to identify patients using computer-based alerts and EHRs to collect outcome data have the potential therefore to increase personal interest in point-of-care trials as an underpinning motivating factor in future decision-making processes.

Patients

Patients should feel that it is their responsibility to help research for the general good by allowing their records to be used. But will the admission that the GP does not always know which treatment is best affect the faith of patients in their doctor? This may not inspire confidence.

It should at least be possible to ask all patients attending a GP surgery for a routine appointment if they are ‘willing to be contacted’ about possible research projects. Consent could be signified by pressing a button at the appointment monitor point, and then flagged on their EHR. Opportunities to broach the subject of research (e.g. annual check-ups for the chronically ill; ‘well-woman’ clinics; and dedicated ‘flu jab’ clinics) could all be used to capture permission. Waiting room posters or rolling TV advertisements could also encourage bored patients to pick up a leaflet to read while waiting.

When a research opportunity arises, those flagged could be sent a Patent Information Sheet through the post to read before their next appointment. This would give them the required ‘thinking time’ to decide whether or not to take part. Appointment reminders sent by text could also be used to remind patients of the leaflet.

Most medical problems resolve naturally, are self-limiting and do not result in a further visit to the surgery. If a patient does not come back, the doctor and his EHR will have no record of the result of the consultation. It will not be known if the patient cashed any resulting prescription, if they took any of the medicine, or if they completed the course. Patients are rarely asked ‘How did you get on with what I prescribed last time?’. Any comment from the patient is seldom noted on the EHR. When reporting ill effects of medication, the reaction is sometimes ‘That is nothing to do with the medicine’, even if it is mentioned as a possible side effect in the Patient Information Leaflet with the medicine. This may mean that EHR research is limited in practice to the chronically or seriously ill patients who attend GP surgeries frequently or regularly.

Patients participating in a trial of commonly prescribed medicines should not be put in the position of having become accustomed to a trial medicine and then being told they are not allowed to continue with it if they prefer it. However, they could be asked to try a different drug after a wash-out period, to see which works best for them. Likewise, if a GP prescribes a branded drug rather than a generic because the patient reports that the generic does not work as well, this decision should be respected, irrespective of financial or practice target considerations. Within reason, individuals should have the right to choose.

Special efforts may need to be made to involve those with poor literacy skills and those who are hard to reach who attend surgeries rarely. Ethnic minorities, who may be the majority of patients in certain practices and may need forms translated or an interpreter, should not be excluded and deserve to be helped to contribute to research If they wish.

Incentives

In general, altruism is the best incentive for a patient to take part in research. Attendance at a manufacturer’s trial may be rewarded with a £10 shopping voucher. However, cash or tokens are unlikely to aid recruitment unless of a significant amount, which only commercial researchers are likely to be able to afford. Anyone to whom a small amount is significant could be considered vulnerable to coercion.

However, if extra visits are involved, travel expenses should be offered as a matter of course, (and not need to be asked for, as this can be thought of as demeaning). Parking places should be provided and/or paid for, and child or elder care likewise if this is a barrier to participation.

General practitioners

General practitioners personally should also feel bound to take part if their practice will not be adversely affected by extra work and appointments, and unreasonable demands on their patients. If protocols are kept simple and records are captured electronically, with no extra work involved, there can be little reason to refuse to investigate further, even if the eventual decision is not to participate. But the ordinary appointment schedule should not be delayed by explanations of research. Some willing participants may not be able to spend extra time with the GP such as those with children to meet from school, babies to feed, seniors to care for, or those without access to their own transport.

Practices taking part in research should not be penalised by the cost of the trial drugs, or the effect of choices on NHS practice performance targets.

Ethics Committees and research and development

It is illogical that a doctor can prescribe whatever he thinks best, but as soon as it is called research, he has to jump through hoops. Information sheets and consent forms for patients should be brief and use clear simple language with a target reading age similar to that of a red-top national newspaper, with no acronyms or technical terms. If successful, the answer to the question ‘Is there anything you do not understand or would like explained further?’ should be ‘No, thank you. That is all very clear’. This should minimise the time needed for explanation and seeking consent. The standard Patient Information Sheet is not suitable for simple EHR research and Ethics Committees should not ask for it to be in that format.

Delays caused by Ethics Committees and R&D offices are not confined to EHR research. An application should not be needed for each individual GP practice or site involved. Approval could also be granted for all EHR research topics at that site in future. The longer the start of research is delayed, the more likely a practice is to lose enthusiastic skilled staff or staff who have possibly undertaken specific training.

Information technology systems

Practices use a number of different IT systems, and have different degrees of access to their host network and even to their ‘desktops’. Knowledge, skills and enthusiasm of staff for IT is also variable. Much time can be spent getting systems to work together. To limit delays, participation could be limited to practices where similar problems have already been overcome and set-up is judged likely to succeed without an inordinate involvement of skilled support staff.

Data sharing

Out-of-hours practitioners, dentists and mental health professionals should, as a matter of course, be able to access patient EHRs, both to read and to enter information. Indeed, from a patient viewpoint, data sharing should be one of the main purposes of EHRs. However, records of referrals, results of hospital tests or emergency care, etc., do not always appear promptly, or at all, on a patient’s EHR, and this could worsen when Social Services data is integrated into the same record at some point.

The use of EHRs in future pragmatic point-of-care trials would therefore be facilitated by:

• fast-tracking the ethics and R&D approvals, particularly for multiple sites within a Commissioning Group or area. The longer it takes to get research up and running, the more likely it is that practices will lose the staff committed to carrying it out.

Choosing GP practices, where:

• IT systems are compatible

• all GPs are enthusiastic, ensuring a common approach whichever GP is seen by the participant

• and data recording can be shown to be generally robust and of good quality.

Choosing topics for research:

• with input from GPs

• with no implications for NHS targets or NICE restrictions

• and where the data loop will usually be closed by another visit.