European network for Health Technology Assessment Joint Action (EUnetHTA JA): a process evaluation performed by questionnaires and documentary analysis

Project structure

The project was divided into eight work packages co-ordinated by a secretariat:

• co-ordination

• communications

• evaluation

• common core HTA

• adapting existing HTAs from one country into other settings

• transferability of HTA to health policy

• monitoring development for emerging new technologies and prioritisation of HTA

• system to support HTA in member states with limited institutionalisation of HTA.

Thirty-four associated partners (contributing to the budget and receiving a share of the grant) and 30 collaborating partners (participating at their own expense) contributed to the project. A 3-year work plan was devised and followed. 8 Governance was through a steering committee (consisting of all associated partners) and an executive committee (consisting of all work package lead partners).

Project deliverables

The main project deliverables were practical tools for conducting HTA. The two tools of potentially immediate use to most HTA agencies were:

1. The core model: This is intended to serve as a platform to aid co-operation in developing a new HTA report. 9 It describes a number of domains (e.g. clinical, economic, etc.). The underlying idea is that different HTA organisations can prepare each domain then enter their findings into a central library. Each individual national organisation can then prepare its own local report for its own health economy by drawing mainly on material contained within the central library (with minor modifications to fit local idiosyncrasies).

2. The adaptation toolkit: This toolkit helps HTA practitioners convert a report between different health-care settings by working through a number of domains. 10,11

Evaluation of the European network for Health Technology Assessment 2006–8 project

Internal evaluation of the EUnetHTA project was an essential requirement of the EU and was the subject of work package 3. This had three objectives:

1. to provide an audit function during the project with regular feedback to the European Commission and the project organisation

2. to evaluate changes over time during the project period to show development towards the establishment of an effective and sustainable network

3. to summarise lessons learned to support the effectiveness and sustainability of the network in its next phase, from 2009 and onward.

The prospective evaluation consisted of three approaches: annual surveys of project participants, twice-yearly interviews with work package leads and documentary analysis of work packages. 12 It concluded that the project had been successful in developing tools that describe a standard for conducting and reporting HTAs and this should facilitate greater international collaboration. Support was evident for a future network. 12

The evaluation report included nine recommendations for a future sustainable network:

1. Secure funding, and maintain a dedicated co-ordinating secretariat.

2. Improve efficiency through an organisational structure made up of work packages managed by a core of dedicated partners, with less committed partners taking part as a wider review group.

3. Continue developing and evaluating the tools as necessary.

4. Involve people in the work to ensure commitment, a high level of knowledge and a broad basis for decision-making processes.

5. Encourage collaboration and communication among all parties to ensure coherence within groups and EUnetHTA.

6. Continue developing the communication platform and functionality of the clearinghouse to make EUnetHTA a central reference point for HTA in Europe.

7. Arrange face-to-face meetings, particularly at the start of group or committee work to strengthen social coherence and reach a common understanding of the work.

8. Evaluate the tools used in real setting and the technical communication platform.

9. English should continue to be the main language.

Rationale

Evaluation has been defined as, the ‘systematic assessment of the operation and/or the outcomes of a program or policy, compared with a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy’. 28 Such critical reflection can be performed retrospectively (after the programme has ended) or prospectively (designed at the start of the programme). A prospective methodology is the gold standard for evaluation research and was used to evaluate the EUnetHTA JA.

Project evaluation allows monitoring of the processes of the project and achievements against specified criteria for success. This enables assessment of the effectiveness and achievements of the project and the formation of ‘lessons learned’ recommendations to inform future projects. It also ensures accountability against project plans. 29 There are three main types of evaluation for projects:29

1. Formative evaluation: This ongoing evaluation starts early in the project and assesses the nature of the project, the needs that the project addresses and monitors the progress of the project. It identifies gaps in the content and operational aspects.

2. Process evaluation: This monitors the project to ensure it is being completed as designed and to the time schedule.

3. Summative evaluation: This is an overall assessment of the project’s achievements and the effectiveness of its processes. It is completed at the end of the project and provides evidence to support the performance of future projects.

The first stage in the evaluation process was creating a project evaluation plan. The main purpose of the evaluation was to identify to what extent the individual work packages enabled the EUnetHTA JA to meet its objectives. This included project participants’ and external stakeholders’ perception of the project processes and deliverables. Documentary review enabled identification about whether or not the deliverables had been produced by the work streams according to the work plan.

This evaluation was a systematic data collection designed to develop generalisable knowledge and to contribute to quality improvement of the EUnetHTA JA project:

1. The impact of the project was assessed by an outcome evaluation (identifying the success of delivering the stated project deliverables). However, it should be noted that, although it was possible to measure whether or not the outputs had been delivered in accordance with the work plan, assessment of their quality and cost–utility was beyond the scope of the evaluation agreed by the EU Commission.

2. The effectiveness of the project was evaluated by its processes (identifying the effectiveness of the processes employed during the project).

Evaluation questions

There were two primary evaluation questions:

1. Will the EUnetHTA JA achieve its overarching objective, and ultimately did it?

   1. The overall objective was defined in the technical annex for the EUnetHTA JA as, ‘The overarching objective of the JA, including work on relative effectiveness of pharmaceuticals, is to put into practice an effective and sustainable HTA collaboration in Europe that brings added value at the European, national and regional level.’

   2. This was assessed by whether or not an effective and sustainable HTA collaboration, not reliant on project funding, had been set up at the end of the EUnetHTA JA project.

2. Will the EUnetHTA JA achieve its specific objectives, and ultimately did it?

   1. Three subobjectives were defined in the technical annex of the grant agreement for the EUnetHTA JA13 as:

      1. – Development of a general strategy and a business model for sustainable European collaboration on HTA.

      2. – Development of HTA tools and methods.

      3. – Application and field-testing of developed tools and methods.

      • Documentary analysis was used to determine whether or not the work packages had produced their deliverables by the end of the EUnetHTA JA project.

      • A prospective evaluation strategy mapped out and evaluated how the activities and deliverables of the individual work packages supported the EUnetHTA JA objectives. The individual work packages within the EUnetHTA JA can themselves be seen as individual projects (with their own objectives, milestones and deliverables) contributing to the overall programme of work of the EUnetHTA JA. This means it was necessary to aggregate evaluation outcomes from the individual projects.

Questions were included in the questionnaires sent to participants to examine:

• demographic information about the nature of participants, their organisations and the HTA information they produce

• the setting-up process for the project

• progress of the project in the interim year

• administrative support and communication from the co-ordinating secretariat

• the role of information technology in supporting the project

• involvement of external stakeholders

• the workings of the eight internal work packages

• the planned follow-up project – European network for Health Technology Assessment Joint action 2 (EUnetHTA JA2).

Questions were included in the questionnaires sent to external stakeholders to examine:

• the mechanisms of the stakeholder forum

• the EUnetHTA JA project

• the workings of the eight internal work packages

• the planned follow-up project – EUnetHTA JA2.

Various terms were used in the evaluation and these are summarised in Box 1 .

Evaluation of the processes of the European network for Health Technology Assessment Joint Action project

A mapping exercise was undertaken to identify which aspects of the project should be assessed at the different time points. One of the objectives of the evaluation was to ‘generate maximum support for the work packages whilst actively seeking evaluation and monitoring solutions that minimise the burden’. This objective was met by decreasing the burden of answering extra questionnaires issued by different work packages. This was achieved by combining questions from other work packages into the questionnaires. The evaluation had been specifically contracted to include questions from the information management work package (WP6). As a courtesy it also included questions from the training strand of the WP8 work package in all three questionnaires and from the dissemination work package (WP2) in 2010. Extensive collaboration was undertaken with the leaders of these other EUnetHTA JA work packages to incorporate their questions. In addition, a questionnaire design workshop was held with participants of the WP6.

Evaluation was performed at three checkpoints within the project, i.e. baseline, interim and towards the end:

• The formative evaluation started at the baseline of the project (month 6, June 2010). This incorporated the key processes of measuring and monitoring to enable identification of emerging issues, and feedback to the executive committee by evaluation reports. These reports were an appendix to the annual reporting mechanism. The baseline questionnaire captured expectations for the project, experiences during the set-up and identified early concerns about the project.

• The interim evaluation (mid-term evaluation) of progress against the project plan was performed in month 18 of the project (June 2011). This identified progress against the plan and identified problems requiring corrective action, which were fed back to the executive committee.

• The final annual questionnaire survey was performed in month 30 (June 2012). This was a towards-the-end evaluation to identify whether or not the objectives of the EUnetHTA JA had been met and flagged-up problems effectively resolved. A limitation of the evaluation was that the EU had requested this be completed before the end of the project.

Evaluation of the plenary assembly meetings

Following the process described above, a standard paper evaluation form was designed to measure participants’ attitudes about attributes of this policy-setting meeting. This was designed to consider:

• whether or not the meeting met its objectives

• satisfaction with the conference venue and facilities

• what the best thing about the meeting was

• what the worst thing about the meeting was

• how the next year’s meeting could be improved.

Analysis of the ‘open’ questions from the 2010 survey allowed identification of themes. Therefore, the 2011 and 2012 surveys also included a grid of meeting attributes and Likert answer options to address these themes:

• receiving meeting documents in advance

• leadership of the meeting

• relevance of items discussed

• meeting and networking with colleagues

• venue and meeting facilities

• social event.

At the request of the secretariat, additional questions were included in the 2011 and 2012 surveys. These addressed:

• when meeting documents were received

• when meeting documents were read

• input into developing the meeting agenda.

Evaluation of the processes of the European network for Health Technology Assessment Joint Action project

It was feasible to survey the entire population and, therefore, no selected sampling was required. An Excel database was obtained from the EUnetHTA JA secretariat which contained details of the EUnetHTA JA project participants in the individual project work packages. This was restructured into a comprehensive database of all the project participants using the following data fields:

• first name

• surname

• e-mail address

• organisation

• membership of individual work packages.

The heads of each organisation were asked to confirm the validity of the information of their staff working for EUnetHTA JA. This feedback resulted in a fairly substantial refinement of the contact database.

A pre-notification e-mail was sent to all project participants in advance of sending them a questionnaire. This enabled the correction of any incorrect information and querying of e-mails for which ‘bounce backs’ were obtained. Pre-notification of survey recipients prior to the questionnaire send-out is a strategy that has been shown to increase response rate by 50%. 40 It emphasises the legitimacy of the questionnaire and communicates the value of response. 43 For the current study each recipient of a questionnaire was sent an e-mail 1 week before the send-out notifying them when the questionnaire would be sent to them, the importance of their completing it and the deadline for completion. This also served the purpose of checking e-mail addresses and correcting any errors about work package membership (in the case of project participants) or sending the survey to a nominated colleague instead (in the case of stakeholders).

Details of the representatives of the stakeholder organisations were also obtained from the EUnetHTA JA secretariat.

Evaluation of the plenary assembly meetings

All participants who attended a meeting were surveyed. Therefore, it was unnecessary to perform this stage because the recipients were physically in the room.

Evaluation of the processes of the European network for Health Technology Assessment Joint Action project

Questionnaires were piloted with members of the lead partner organisation who were non-native English speakers. Comments were also sought from members of the EUnetHTA JA executive committee and the European Commission. A design workshop was held with members of WP6 in 2010 and the lead partners of WP6 (French) and WP8 (Spanish) also piloted the questionnaires prior to the send-out.

Evaluation of the plenary assembly meetings

The form was piloted by members of the lead partner organisation.

Evaluation of the processes of the European network for Health Technology Assessment Joint Action project

The choice of mode of transmission of the questionnaires in the current research was between postal mail, telephone, e-mail and web based. For this study the participants were geographically dispersed over Europe and, therefore, postal distribution was not a practical possibility owing to financial and logistical limitations. Telephone questionnaires were also impractical. A web-based questionnaire that could be completed at the convenience of the respondent was preferable and the recipients were familiar to e-mail communications and using the internet.

Web-based surveys offered several advantages compared with post for the current research: there was a significant cost reduction when considering the international survey population and a quicker turnaround time. 44 In addition to electronic transmission of surveys by e-mail, web-based applications enable automatic data collection. The design of web surveys may be more important than for print surveys. This is primarily because a web-based survey can be displayed differently to a respondent as a result of computer-related glitches and the coding system offers more design capabilities than print. 44 However, the increased ability for design capabilities must be used with some caution because too many design features may lead to overcomplication and a decreased response rate.

Different possible online survey platforms were investigated for the distribution of the questionnaires and SurveyMonkey.com^® (Palo Alto, CA, USA) was selected for ease of use and function capability. A paid, professional account was selected from SurveyMonkey.com. 45 This enabled:

• an unlimited number of questions and responses

• a custom uniform resource locator (URL)

• a branded survey with a logo – use of the EUnetHTA logo and colours

• a survey completion progress bar

• a custom redirect on survey completion – to the eunethta.net project page

• a printable PDF version for sharing during design collaboration

• importing e-mails into the ‘survey manager’ send-out function

• easy tracking of non-respondents and distribution of follow-up e-mails.

Care was taken that the questionnaire invitations were not sent during the weekends and that the public holidays in the different European countries were avoided. As far as possible these invitations were not sent during the summer holiday season (particularly avoiding the month of August).

Evaluation of the plenary assembly meetings

The analysis of the annual EUnetHTA JA plenary assembly meetings was performed by anonymous self-completion paper questionnaires. These were included as part of the agenda pack given to the meeting participants. They were reminded to complete the questionnaire and hand it in at the end of the meeting.

Evaluation of the processes of the European network for Health Technology Assessment Joint Action project

The recipients had been given a guarantee of confidentiality but not anonymity. This meant that through using the SurveyMonkey.com online platform it was possible to send targeted reminder e-mails to non-respondents. These were personalised e-mails that contained a personal weblink to the questionnaire. E-mails were designed to include two strategies that have been shown to be effective in increasing response rate: each e-mail included a statement that indicated others had responded and provided a deadline for response. 40 Reminders were generally sent 1 week after the date requested for the questionnaire completion and non-respondents were requested to complete the questionnaire within 3 weeks. Two follow-up reminders were sent at 3-weekly intervals.

Evaluation of the plenary assembly meetings

Survey response was anonymous and there was no possibility of tracking non-respondents. Therefore, it was not possible to follow up non-respondents.

Use of tools

These results were obtained from a closed question included in all three of the questionnaires.

Unsurprisingly, the most often used tools were those for project communication – the MO intranet website, workrooms and e-meetings. Not all members used the project intranet site (the MO website), although the proportion of respondents using it increased from 2010 to 2012. However, the missing proportion said they were aware of it and are likely to use it in the future.

Workrooms were areas assigned within the project intranet for the individual work packages. Although approximately three-fifths of respondents used them, they rated them poorly. It is, therefore, suggested that their use is helpful, because they were used by a large proportion of participants. However, there is obviously an important need for them to be improved in EUnetHTA JA2. Workroom bulletin boards were less often used, with only approximately 15% of project participants reporting use. It seems that respondents used the website, but not the subsplit workroom provisions. Further evaluation needs to be performed in EUnetHTA JA2 to identify whether these workroom bulletin boards could be useful if they were improved or if participants considered them superfluous and they communicate by other means, such as e-mail.

There was increasing use of the following other communication tools from 2010 to 2012 (although they were still only used by less than half of participants): contact database (a database of details of all the project participants), EUnetHTA toolbar (used on the internet) and mailing list (a list of all project participants).

Unsurprisingly, the results for use of the HTA tools were significantly lower. This can be explained because most of the tools were still in a development stage, even in summer 2012 (which again illustrates the limitations of performing evaluation before the end of the project). The HTA Core Model was the tool that the most participants had used, and the proportion that had used it doubled during the lifetime of the project to three-fifths. In each year, approximately 90% indicated they had used it or might use it in the future. The adaptation glossary and adaptation toolkit were under development, and this is reflected in the results of limited use. Although being developed as part of the project, the EVIDENT database had been used in the previous project (as EIFFEL) and, therefore, a number said they had used it. This database was seen as being relatively less popular because approximately 60% saw a future use for it. The POP tool was converted from a static datasheet to an interactive site during the project and this was reflected in the results; the proportion of participants using it doubled to 50%. Roughly 80% had either used it or saw future use for it.

Priority for training

These results were obtained from a closed question included in all three of the questionnaires.

The HTA tools were consistently seen as being a higher priority for training throughout the lifetime of the project than the project communication tools (which were mostly cited as low priority or of no importance). Therefore, the ‘content’ tools were seen as of greater importance than the ‘process’ tools of the project. Possible reasons for this include that there was greater need within the project for more methodological tools, that such type of tools are more complex to use or that communication tools were easy to navigate. In each year the tool most often cited as ‘top priority’ for training by project participants was the core HTA model. The importance of training both partners and stakeholders in EUnetHTA tools and methods was emphasised in the plans for the EUnetHTA JA2 project. One of the deliverables of the JA2 project is ‘Report on yearly training courses on EUnetHTA tools and methodology’. It is planned that three face-to-face training workshops will be held on EUnetHTA tools and methodology. It is also planned that three additional training courses will be held for the core HTA model. These plans were proposed as a result of evaluation questionnaire findings of EUnetHTA JA.

Training method

These results were obtained from a closed question included in the 2010 and 2012 questionnaires.

In general, the most preferred training method for tools was self-directed with a manual. However, a ‘face-to-face workshop’ was the preferred training method for the HTA Core Model, with over half of respondents preferring this option. As a consequence of the 2010 survey results, a face-to-face workshop about the HTA Core Model was organised as part of the project. It was interesting that the proportion of respondents requesting a face-to-face workshop about the EVIDENT and the POP databases noticeably increased from 2010 to 2012. It is difficult to conjecture the reason for this, but this could have been in part due to satisfaction of respondents with the workshop they had attended about the HTA Core Model. One advantage of using a manual is that learning is self-directed and can be undertaken at a time and place to suit the convenience of the learner. Self-directed manual use is also a cheaper method and negates the need to travel to attend a face-to-face workshop; however, it has the disadvantage that queries and difficulties using a tool cannot be as easily resolved as they could be at a training workshop.

Barriers to using the tools

These results were obtained from a closed question included in all three of the questionnaires.

Respondents were asked whether or not any barriers had prevented them using the tools from ‘organisational’, ‘training’, ‘the tool itself’ and ‘IT’. Encouragingly there was an overall general trend that the frequencies of all barriers decreased during the lifetime of the project. This may have been because problems were resolved during the project – informally or formally. There was a general trend that the effect of training being a barrier to tool use decreased during the project throughout the lifetime of the project from 2010 to 2012. It is possible that as the project progressed, participants picked up knowledge about using the tools either informally or formally (i.e. by the training course for the HTA Core Model). IT was seen as a significant barrier to the performance of e-meetings and training was a noticeable barrier to the HTA Core Model (and is reflected in the decision to hold face-to-face workshops in EUnetHTA JA and EUnetHTA JA2). When asked in 2012 why they had stopped using a tool, few responses were obtained. Reported problems included tools not yet being active, problems with passwords and IT problems with e-meetings.

Project start-up

These results were obtained from a combined closed and open question included in the 2010 questionnaire.

Approximately three-fifths of participants were satisfied with the set-up of the project, but one-fifth thought it could have been better. Suggestions included better transparency, decreased bureaucracy, better communication from the JA secretariat and the EU Commission, greater preparation time and maintenance of constant personnel dealing with the project.

The initiation stage of a project is crucial. The most important factor is gaining consensus from project participants about what is to be achieved, and how this should be done, at the outset. 55 A well-planned project will avoid scope creep further down the line by defining the purpose, scope, roles, costs and schedule.

Organisation into work packages

These results were obtained from a combined closed and open question included in the 2010 questionnaire.

In common with other such international projects, the EUnetHTA JA project was structured by division into eight distinct work packages. The majority of participants and stakeholders who expressed an opinion thought this structuring was positive, but approximately one-quarter of participants thought there would also be negative aspects associated with this division of work. Concerns centred around possible overlap between the aims and work of individual work packages and the importance of effective communication between them. Most projects funded by the EU are formed in this structure of individual work packages that can be viewed as discrete projects. As one participant responded, ‘it is difficult to see how else the project could have been arranged’. However, this structure means that communication was required between the individual work packages to avoid duplication of work and, wherever possible, enable collaboration. This was done by bimonthly meetings of the executive committee, which included the leaders of all work packages along with the secretariat and the representative of the European Commission. Such a steering committee is recommended for complex, international projects. These committees prevent tension between the individual projects and allow for greater flexibility at the operational level. 58 In their consideration of international project management Lientz and Rea considered having two committees: an overseeing committee and one composed of more senior work package leaders. 55

Sufficient resources

These results were obtained from a combined closed and open question included in all of the three questionnaires.

Participants provided broadly similar results between 2010 and 2012, with only about a half indicating that their organisation had sufficient funding and three-fifths indicating that there was sufficient staff. The proportion that thought they did not have enough resources was fairly constant at approximately one-fifth over the 3 years. A large number of participants indicated that they did not know whether or not their organisation had sufficient funding or staff to fulfil their obligations to the EUnetHTA JA project. This may be because not all respondents were senior enough to know details about financing, but a lack of staff might be more apparent to them. In hindsight, it might have been better to have asked this question to the organisational lead to obtain views from only senior staff (although this might be biased).

It is obviously important that organisations had sufficient staff and resources to be able to fulfil their commitment to the EUnetHTA JA. From the data collected it is unclear whether it was actually the case that organisations did not have sufficient resources or this was the perception of their staff. Implications of insufficient funding cited by project participants were that the work required extra time for completion than had been estimated or organisations were subsidising the work themselves by working at weekends, devoting more staff to EUnetHTA than were paid by project funding. Lack of staff meant increased workload within the team, and this could have a negative effect on staff morale. For an international project it is important to realise the importance of the self-interest and resources of each of the individual organisations at the start. 55 It is important that the individual participant organisations feel that they will benefit from the project. It cannot be assumed that organisations will contribute their best resources or effort to the project, to make up any shortfall. This can mean that the work was conducted to a basic standard, with the project work being viewed as a non-priority task. Conflict with normal work of a project participant is also an important factor. Most international team members are not solely dedicated to the project, but must fulfil requirements of their regular job in addition. 55 This could also lead to decreased enthusiasm within an organisation for participation in future international projects. This seemed to be particularly important in this project because the funding was obtained in a 50 : 50 ratio from the EU and the member state.

Cultural factors and different styles of work in an international project should also not be overlooked. Business needs are diverse in each country involved in the project. The project should address such local issues otherwise it will be seen as negative – requiring resources but not delivering benefits. 55 Also, in the EUnetHTA JA the individual work packages can be seen as inter-related individual projects. It is important to realise that these projects can share the same resources and together can exact too high a stress level from workers. 55

Difficulty joining the project

These results were obtained from a combined closed and open question included in the 2010 questionnaire.

Hearteningly, less than one-tenth of respondents reported a problem with their organisation joining the project. Problems reported included lack of a plan, lack of transparency and lack of distinction in work between performed in the preceding collaboration year and the start of the EUnetHTA JA project. It is important that the process for organisations joining a project is straightforward and easily manageable. Almost one-quarter of project participants in 2010 had a suboptimal or mixed understanding about the information that was required from the secretariat before the project started. The proportion who understood requirements increased from before the project began (two-fifths) to after the project started (three-fifths). Concerns included that the information was not always clear and recognition about the complexity of the project. The relative scarcity of problems can be seen as a good indicator of success.

It should also be borne in mind that the number and types of organisations within the EUnetHTA was a fluid situation, with organisations able to leave and join during the project. It would be useful to have a type of induction process for organisations that joined part-way through the project. It could also be useful to produce a map of the characteristics and expertise of the different organisations joining the project, to enhance collaboration opportunities within the network.

Succession planning

These results were obtained from a combined closed and open question included in the 2010 questionnaire.

Almost two-fifths of organisations had no succession plan in place in case a member of staff became ill or left the organisation, with a further one-fifth not knowing whether or not they had one. It is vital that there is a plan in place in case members of staff leave, to ensure that this does not affect the work of the project. Loss of team members can mean that experience and knowledge is lost and activity in an organisation can grind to a halt. 55 In combination with the fluid nature of high turnover of project staff, there was an apparent lack of succession planning. This again reinforces the need for communication and induction of new staff. Examples of this included the secretariat administrator going on maternity leave and the individual leading the contribution of the lead partner of WP2 leaving mid-way through the project. For an international project, there needs to be a local commitment that the project team is stable and committed, and members are not pulled off the project by local management to perform other work. 55 However, in practice little can be done to mitigate the effect of individuals leaving to take up other jobs, going on maternity or sick leave or dying.

Challenges

These results were obtained from an open question included in all of the three questionnaires.

Qualitative comments from participants about the challenges inherent in the project included the large scope of the project, its international nature, high workload, limited time and resources, the work itself, communication, administration, national/organisational conflict, requirements of individual work packages, imbalance between contributions of participants and problems with collaborative working. A specific problem in 2010 and 2011 was orientation within the project.

After analysis of the emergent themes from the P2010 results, the prevalent project challenges were categorised into a quantitative question using a three-point Likert scale. A large effect was assigned to conflict with other work activities and insufficient organisational funding and staff. Combining the frequencies for the effects showed that approximately three-quarters were affected to some degree by conflict with other work and insufficient staff and funding. The effects of insufficient organisational funding and staff, and conflict with other work have been discussed in earlier sections.

The effects of the large project scale and demands of the individual work packages were small. Interestingly, the demands of individual work packages seemed to increase between 2011 and 2012. The large project scale is one of the inherent problems of international projects and this leads to a complex nature of subprojects. A large proportion of project participants were members of more than one work package, which meant they had to juggle various work demands.

Hearteningly, more than half of respondents experienced no effect from difficulty in communicating in English and difficulties in communicating generally. Communication is affected by various types of culture and it can be difficult to manage different styles in an international project. 58 In addition, project participants had various natural and functional languages. In the EUnetHTA JA project it was necessary to use a common language, which was English. However, it must be considered that the use of a language by non-native speakers can result in insufficient project feasibility, loss of speed, loss of creativity, poor decisions, underuse of resources and unexpected misunderstandings. It is possible that such problems were experienced by participants, but were not deemed sufficiently important to affect the project work significantly.

Secretariat leadership

These results were obtained from a combined closed and open question included in all of the three questionnaires.

Approximately three-fifths of respondents thought the secretariat had offered effective leadership. The number of respondents who thought the leadership was OK (but could be better) increased from less than one-tenth in 2010 to one-fifth in 2012 suggesting an apparent slight decrease in confidence as the project progressed. The qualitative data revealed that concerns included negative aspects relating to communication from the secretariat, implications of changes in the secretariat staff, recognition of the difficulty of the task, a distant style used, and a lack of direction provided by the secretariat and that this needs to evolve in the future.

Important attributes for leaders of international projects have been defined. 55 These include problem-solving ability, ability to cope in multiple cultures with diverse political problems, tenacity and the capability of pursuing issues, ability to communicate, a sense of humour, familiarity and knowledge of the business and prior experience in projects. It is also important that the project leader is aware of the intra- and inter-organisational links in the project and is aware of the hidden agenda of individual organisations.

Owing to the specific nature of the EUnetHTA JA project, it was also important that the leader had a strong steer for HTA in Europe. It has been recommended that there are two leaders for international projects. 58 In the EUnetHTA JA project, as in the EUnetHTA 2006–8 project, the leadership was provided by Finn Borlum Kristensen. The secretariat manager was Julia Chamova. From the comments above, it would seem that the leadership was sometimes perceived as somewhat distant and this could be helped by more regular communication about ‘leadership issues’ with participants throughout the project. There was some concern expressed about the over-reliance on Finn Borlum Kristensen and what would happen if he became ill.

Secretariat administration

These results were obtained from a combined closed and open question included in all of the three questionnaires.

Effective project administration is essential for any international project. Such a co-ordination role can organise the project files and project history, oversee the lessons learned and issues log, and support and mentor the project leaders. As such, the secretariat functioned as the project management office and it had an important role in both the internal project processes (e.g. monitoring the performance of individual work package projects) and connection to the external world (e.g. by interacting with external stakeholders). For the EUnetHTA JA project, as for the EUnetHTA 2006–8 project, this function was performed by the secretariat, based at the National Board of Health, Denmark.

Approximately three-fifths of respondents thought the administration had been effective. The proportion who thought secretariat leadership was OK (but could be better) rose from one-tenth in 2010 to one-fifth in 2012. This indicates that there was a slight increase in dissatisfaction with the administration as the project progressed. The qualitative data showed that concerns included, problems with e-mail, general communication problems, too short timelines, quicker feedback required, problems with work packages and the belief that more staff were needed in the secretariat.

The main method by which the project secretariat communicated with work package leaders and individual project participants was by e-mail. Therefore, participants were asked about their opinions about the nature of this communication. Approximately three-quarters of respondents thought the e-mails from the secretariat were acceptable, in terms of frequency and content. Comments related to clarity, frequency and length, that e-mails had improved from the previous EUnetHTA project, the importance of ensuring relevant addressee and suggestions for improvement (e.g. to use a standard template and include the work package number and deadline in the subject line). Although e-mail is a practical mode of communication, it is low down the hierarchy of communication methods because of its inherent problems. Suggestions have been made to improve e-mail communications in international projects. These include using specific titles, making the important point(s) in the first six to eight lines and avoiding overloading participants with e-mails. 55

Additional support from the secretariat

These results were obtained from a combined closed and open question included in the 2010 and 2011 questionnaires.

Approximately one-fifth of respondents in 2011 thought there were other activities that the secretariat could do to support the project, and this proportion had risen from 2010. Suggestions included facilitating relationships, providing greater feedback, providing advice about monthly budgeting, being more customer orientated, facilitating development of effective tools, facilitating information exchange, providing project management and enabling effective communication.

Communicating in English

These results were obtained from a combined closed and open question included in all of the three questionnaires.

In any international project, communication is likely to be difficult because all members do not share a common language. It is important that there is a common language for communication. 58 Encouraging, approximately three-quarters of respondents had not experienced any significant problems when communicating in English during the project. The qualitative data showed that concerns included recognition of the inherent nature of communication problems, difficulty with English text, difficulty with audio, difficulty with speech, the need for greater time when communicating in English, recognition that language barriers exist, differences between nationalities, causes problems during meetings, difficulty in finding staff with good English-language skills and the suggestion of checking of documents by native English speakers. It should also be considered that there can be cultural aspects of communication. For example, Scandinavians and north Europeans tend to use a low-context style, where communication is explicit and unambiguous. In comparison in high-context styles (typical of southern Europeans), meanings of words can be hidden. 58

Communication methods

These results were obtained from a combined closed and open question included in the 2011 and 2012 questionnaires.

Owing to the large and complex structure of the project, it was essential that communication was optimal. Participants working on tasks in the individual work packages were examples of virtual teams, ‘. . . teams of workers who are dispersed across geographical, temporal, and organizational boundaries, yet collaborate using information and telecommunications technology’. 55 To help facilitate this, a large number of different types of communication methods were used. Of these, the apparent most useful mechanism was face-to-face meetings. This method was most frequently described as very useful in all years (although this decreased from three-quarters in 2011 to two-thirds in 2012). In the hierarchy of communication methods, in person communication has been ranked as the gold standard method because it is possible to see a person’s body language and catch the tone of voice and any specific nuances. 55 This method is also the best mode for establishing trust59 and discussing any controversial issues. 58 The observed inclination for face-to-face meetings might indicate that the EUnetHTA JA is a ‘relationship-orientated culture’ as such cultures prefer face-to-face meetings because of the benefits of physical, social and situation context. 58 A preference for face-to-face meetings goes hand-in-hand with the importance of networking and reinforces the importance of participants meeting in person as opposed to working solely in virtual teams. However, the benefits of this communication method need to be balanced with the inherent implications in terms of financial and logistical costs. Therefore, it is important that face-to-face meetings are conducted in an optimal manner and this demonstrates the importance of evaluating the policy-setting annual plenary assembly meetings.

Other popular methods for communication were by using the project intranet (the MO website) and by secretariat e-mails. These e-mails were a synchronous mode of communication, in that all (relevant) participants received the information at the same time. This is a communication better suited to task-orientated communications. 58 The MO website acted as the project intranet and was a central repository of project information. A large proportion of members did not know about the effectiveness of the plenary assembly. This was to be expected, because only one representative per organisation was permitted to attend this meeting.

Participants considered that communication could be improved in the EUnetHTA JA2 by having a formal communications plan, devoting more resources to it, improving the current systems (e-meetings, secretariat, website), using social networks, online tools, expanding networking opportunities, using project management techniques and improving communication with stakeholders. Stakeholders mostly considered this from the perspective of their receiving more information themselves rather than communication between workers on the project. Social interactions and networking is especially important for geographically dispersed teams. 59 In this respect, use of social networks such as Facebook (San Francisco, USA), Twitter (San Francisco, USA), LinkedIn (San Francisco, USA). may be useful. This point will be further investigated in the EUnetHTA JA2.

External promotion

These results came from an open question included in the 2012 questionnaire.

For projects generally little attention has been given to project marketing, which is concerned with communicating the long-term consequences of a project. The relationships between internal and external stakeholders are important. Suggestions for improvement included the need for a greater presence at conferences, need for more published information, improvement of the public website, national relevance should be communicated to countries, a need to target relevant stakeholders, social networks should be employed, missing an individual approach, more training needed and that promotion will be demonstrated best by collaborative HTA reports. There should be a distinction between whether the project has been insufficiently promoted externally or if that is a perception. This workstrand was the subject of the dissemination work package, WP2. Detailed comments about the functioning of this work package are described below.

All work packages

All project participants, both members and non-members, were asked general questions about the individual work packages. External stakeholders were also asked for their opinions about the work packages.

Notably concern was expressed from the participants about the tool-generating work packages: WP4 (by almost one-quarter), WP5 (by almost one-fifth) and WP7 (by almost one-fifth). One-fifth of respondents had concerns about WP8. Almost half of respondents did not know if they had concerns about WP2 or WP3, and it is speculated that the fact these respondents did not know if they had concerns or not indicated a general lack of knowledge about these work packages. In contrast, external stakeholders had a higher rate of concerns about work packages which were not associated with developing HTA tools, i.e. WP1, WP2, WP3, WP6 and WP8.

This apparent concern about HTA tool-generating work packages by participants was apparently not reflected in whether or not participants thought their objectives would be met. However, for most work packages over half of the respondents did not know whether or not the objectives would be met. It is possible that respondents who were not part of a specific work package did not know whether or not the objectives of other work packages would be met or it could be that they thought the objectives would not be met but were unwilling to provide a negative response. The proportion of respondents who did not know was also very high for the stakeholders, again over 50%. This could suggest a lack of confidence in work packages being able to meet their objectives. Stakeholders showed lack of knowledge about objective meeting by work packages not associated with developing HTA-tools.

Approximately one-fifth of participants thought they had not received appropriate communications from the majority of work packages. However, a higher proportion thought they had not been adequately informed about WP2 and WP8. Two-fifths of stakeholders thought they had received insufficient communication from WP1. One-quarter of stakeholders thought they had received insufficient communication from WP2, WP6 and WP8.

There were minimal proportion of participants who expressed an opinion that work packages were not worth having. However, approximately half of respondents did not know if WP2 and WP8 were worth having in the project.

There was an indication from stakeholders that they had received insufficient communications from work packages in which they were not members of. They requested greater involvement of stakeholders in other work packages (e.g. by stakeholder advisory groups being set up) and involvement of professionals with specialist expertise in dissemination and business development.

Specific work packages

For the following section, project participants who were members of the individual work packages were asked detailed questions about them.

The individual work packages of the EUnetHTA JA can be viewed as individual projects in a portfolio. A virtual organisation is formed of virtual groups of workers. 59 A virtual work place forms a barrier to the richness of face-to-face interactions and for this reason the usefulness of face-to-face meetings have been used to jump-start an effective team. 59 Three factors are important for the performance of successful virtual teams: shared understanding, integration (working together in a way that creates value) and mutual trust. 59 A list of attributes of team members of international projects has been developed, which includes experience in similar projects, previous experience in international projects, ability to work with other people on tasks, ability to solve problems and work within the organisation, sensitivity to issues and potential problems, availability from their other work to perform tasks on the project, communication skills, ambition and energy, ability to cope with different cultures, ability and willingness to travel, and the ability to work hard at understanding and getting ideas across. 55 All project members have a responsibility for building trust, communicating, using and adhering to common ground rules and managing conflicts. 58

Duties of a project manager of an international project include defining the project by dealing with ambiguity and politics, organising the project, dealing with the routine work, addressing issues and crises, leading by example, co-ordinating administration (e.g. logistical arrangements of face-to-face meetings. 55 Trust between project members is of vital importance in international projects, alongside the belief that fellow team members are putting their best efforts into the work and everyone is working towards the project goal. 58 Such trust is one of the prerequisites of managing global teams. This is difficult to establish in an international project because of knowledge of collaborating partners being low and reliance on virtual working practices. 58 It is also helpful to have informal interaction opportunities for each team.

Some of the main sources of conflict in international projects include ambiguity of project objective, insufficient authority of the project manager, manpower resources, costs, equipment and facilities, priorities and responsibilities. 58 One factor that can be important in the on-time delivery of a project is ‘adequacy of documentation of organisational responsibilities on the project’. 57

Work package 1: co-ordination

• There appeared to be a slight depreciation over time as the proportion who agreed increased and proportion who strongly disagreed decreased for leadership and communication being effective within the work package and deliverables being clear.

• The proportion who disagreed increased over time about the objectives of the work package being clear and the amount of work being manageable, which was one-tenth in 2012.

• Approximately one-fifth disagreed that the work package had benefited from the inclusion of stakeholders and one-quarter that all partners had contributed to the work adequately (2012 questionnaire).

• Almost one-third did not know how this work package had progressed in 2011, but this decreased in 2012 where approximately half thought it had been OK and half that it had progressed well.

• The proportion of respondents having concerns about the work package increased over time to one-fifth in 2011. The qualitative data showed that concerns included continuity, high dependency on the lead, too many face-to-face meetings, increased workload, overlap with EUnetHTA JA2 and about the business model.

Overall, this work package appeared to perform well over time with respect to effective leadership, clear objectives and manageable workload. However, concerns included the benefit of stakeholder inclusion and equal contribution of partners. There appeared to be a strong dependency on the lead partner. Concerns about the delivery of the business model and overlap with the EUnetHTA JA2 project have been addressed previously.

Work package 2: dissemination

• The proportion of respondents who disagreed about leadership and communication being effective within the work package increased over time to one-quarter and two-fifths respectively.

• The proportion of respondents who did not know was two-fifths or more for the planning timeline being clear, objectives being clear (P2011), progression of the work package over the past year (P2011), frequency of e-meetings, frequency of face-to-face meetings and whether or not the work package had communicated adequately with other work packages.

• Four-fifths did not know about the benefit of involving stakeholders in the work package.

• One-fifth of respondents thought the progression of the work package was poor. The qualitative data showed that concerns included the need for more communication, internally and externally, and more support from the co-lead partner.

Overall, this work package appeared to have performed somewhat poorly, with respect to effective leadership and communication. Particular concern was about the leadership, communication and benefit from the involvement of stakeholders. It was relevant that the representative from the colead organisation had died and the representative from the lead partner organisation had left part way through the project. This emphasises the importance of continuity of a project and the need for induction processes so that the project can continue seamlessly.

Work package 3: evaluation

This work package only contained one organisation (NETSCC) and two members of staff and, therefore, the evaluation questions were not asked for this work package.

Work package 4: core health technology assessment model

This was one of the core work packages of the EUnetHTA JA, with the aim of producing a tool for performing HTA. Therefore, stakeholders were also asked about how they thought it had performed.

• The proportion of respondents who agreed increased, but the proportion who strongly agreed decreased for leadership being effective, objectives being clear, and deliverables being clear.

• The proportion of respondents who disagreed about communication between members being effective increased over time to one-fifth in 2012.

• Approximately one-fifth of respondents disagreed about the amount of work being manageable, collaboration between members to produce core HTAs and benefit from the involvement of stakeholders. About two-fifths disagreed about whether or not all members had contributed to the work adequately.

• The proportion with general concerns about this work package decreased over time from one-quarter to one-fifth. The qualitative data showed that concerns included the applicability of the model, co-ordination issues, communication issues, workload and that stakeholders had been of little use

• By the final year approximately half of respondents thought it had progressed OK and half that it had progressed well. The qualitative data showed that concerns included slow progress, lack of participation in pilots and would form the basis for EUnetHTA JA2.

• Stakeholder concerns included limited applicability of the model in practice.

Overall, this work package appeared to perform well, with respect to effective leadership, and clear objectives and deliverables. However, some concerns were expressed about the amount of work, collaboration on the work, contribution of members and involvement of stakeholders. This was one of the work package that contained a larger amount of participants, which could have magnified intrinsic difficulties in communicating, dividing the work evenly and working together.

Work package 5: relative effectiveness assessment of pharmaceuticals

This was one of the core work packages of the EUnetHTA JA, with the aim of producing a tool for performing HTA. Therefore, stakeholders were also asked about how they thought it had performed.

• There appeared to be some degree of improvement in the leadership being effective and communication between members being effective because one-tenth disagreed in 2010, but this decreased to zero in 2011 (when one-tenth did not know).

• The proportion of respondents who disagreed about the amount of work being manageable increased to approximately one-third and a similar proportion disagreed about the contribution of staff.

• Almost one-fifth disagreed about the division of work and one-fifth disagreed about the number of face-to-face meetings being appropriate.

• Approximately half did not know about the benefits of stakeholder involvement, but half agreed involvement of stakeholders had benefited the work package.

• Concerns about the work package’s progress decreased from two-fifths in 2010 to one-fifth in 2011. The qualitative data showed that concerns included that there had been a change in focus from developing a full and rapid model for assessment to only a rapid model, high workload beyond the schedule, applicability concerns, insufficient number of face-to-face meetings, and difficulty in managing stakeholder and political interest.

• The qualitative data showed that stakeholder concerns included lack of involvement, lack of information exchange and inappropriate influence of industry.

Overall, this work package appeared to perform well, with respect to effective leadership and communication. However, there were concerns about the amount of work, division of work, contributions of members and the number of face-to-face meetings (in 2010). This was again one of the work packages with a large number of members, which could cause intrinsic difficulties and the need for face-to-face meetings is highlighted. There was an apparent perception of high workload. It is difficult to conjecture whether there was too much work that had grown from the work plan or if this was members’ perceptions. It is important that workers contribute as equally as possible so that the burden can be shared.

Work package 6: information management system

• There appeared to be improvement over the lifetime of the project, with the proportion of respondents who agreed decreasing and the proportion of those who strongly agreed increasing for communication between members being effective, objectives being clear, the planning timeline is clear and number of meetings was appropriate.

• Approximately one-tenth of respondents thought the technical issues were unclear in 2011.

• Three-fifths of respondents did not know about the benefits of involving stakeholders in the work package.

• Approximately one-fifth of members had concerns about the work package, which included that the work was dependent on the other work packages, the inherited IT system from the previous project and they would appreciate e-meetings to discuss the work between the planned face-to-face meetings.

• The proportion of respondents who thought the project had progressed well increased from approximately one-half to two-thirds in 2012.

Overall, this work package appeared to perform very well, with respect to effective communication between partners, clear objectives, clear planning timeline and an appropriate number of e-meetings. Some concern was shown about the equal involvement of members and of stakeholders. The information technology infrastructure is important for a geographically dispersed project and the other work packages were dependent on it.

Work package 7: new technologies

This was one of the core work packages of the EUnetHTA JA, with the aim of producing a tool for performing HTA. Therefore, stakeholders were also asked how they thought it had been performed.

• There was a noticeable level of disagreement in 2010, which improved slightly over the project; communication between members being effective (one-quarter in 2010), objectives being clear (one-fifth in 2010), leadership being effective, deliverables being clear and planning timeline being clear.

• The proportion of respondents who strongly agreed about the leadership being effective increased, while the proportion who disagreed also increased between 2010 and 2012.

• Three-fifths of participants were unsure about the benefit of involving stakeholders.

• Only about one-third of respondents agreed or strongly agreed that all members had contributed to the work adequately.

• The proportion who thought progress had been good increased in 2011 to three-fifths. The qualitative data showed that concerns included that the two strands had developed quite differently: lack of partners populating the POP database and obscure development of EVIDENT.

• The qualitative data showed that stakeholder concerns included delay in deliverables and lack of involvement of stakeholder advisory groups (SAGs) in the process.

One of the objectives of the process evaluation was to ‘prospectively be vigilant of emerging issues and support the achievement of objectives’. It was identified in the first survey in 2010 that there were concerns about the leadership of WP7. This matter was raised with the lead organisation of WP7 and the executive committee. The WP7 lead partner explained that they had experienced lack of staff to fulfil some of their leadership activities as a result of a staff member going on maternity leave and they had plans to resolve this matter. This was not an apparent problem in the next annual survey of 2011 and so it seemed that the matter had been effectively resolved.

Overall, this work package seemed to progress quite well. After a somewhat disappointing start, it seemed to improve over the life of the project. However, there were some apparent concerns over unequal contribution of partners to the work.

Work package 8: strategy and business model development

• There was still significant disagreement in 2012 of at least one-fifth for communication being effective and the deliverables and objectives being clear.

• Two-fifths did not know if all members had contributed to the work adequately.

• Almost half did not know about communication with other work packages and the benefit of involvement of stakeholders in the work package work.

• This work package was split into three subsections and by the end of the project about one-tenth of respondents for each section thought it had progressed poorly.

• Approximately one-third of respondents had concerns in 2010 about the work package, but this decreased to one-fifth in 2011. The qualitative data showed that concerns included confusion about constitution and structure of the work package into three different work streams.

This was a complex work package and was subdivided into three separate work streams. This was reflected in the concerns expressed about the work package’s structure and may have impacted on concerns about the communication, deliverables and objectives.

Objectives

Participants were asked a closed question in each questionnaire about whether or not they thought the meetings’ objectives had been achieved.

Overall, approximately one-tenth of meeting participants did not know whether or not the meetings’ objectives had been met in 2010 and 2012. About one-fifth were not sure for the 2011 meeting. It is recommended that the meeting objectives are made explicit from the outset. It was concerning that one-tenth of participants thought the meeting’s objectives had not been met in 2011. Causes for concern included questions not being answered, lack of useful information and lack of time for debate. When indicating that the meeting objectives had been met, it appeared that participants cited factors that had been important for them personally, such as information gain and endorsement of decisions.

Most preferred aspect

Participants were asked an open question in each questionnaire about what the best aspect of the meeting had been.

Overall, meeting participants seemed to appreciate meeting people face-to-face, discussions, information sharing, social activity and good meeting logistics.

The academic literature has shown that the agenda is crucial in meetings and it has been recommended that meetings begin with a few easy items to settle participants in. 60 Difficult items should be placed around breaks. Frequent breaks should be factored into meetings because concentration typically wanes after about 45 minutes. This also enables participants more opportunities for informal networking. Larger meetings have been correlated with a lower quality and demonstrated the need for a facilitator. Other attributes that have a positive effect include having a break, sufficient lighting and temperature of the room, room space and refreshments. 61

Least preferred aspect

Participants were asked an open question in each questionnaire about what the worst aspect of the meeting had been.

In 2010 problems seemed to centre around technical problems with microphones during the meeting and difficulties getting to the venue. Good meeting logistics are important for a successful interaction and functioning microphones are an important facet of this. In 2011 there were apparent difficulties posed because of the meeting venue (such as seeing the slides) and chairing of the meeting (such as lack of summing up and apparently treating comments at different levels). It is crucial that tables and chairs are positioned so that the chairperson and all the members may participate appropriately. The academic literature has shown that seating the chairperson separately might help them exert their authority over proceedings60 and it is important that participants are able to express their opinions and everyone is heard, and that discussions are respectful. 61 How the participants viewed the meeting can have important attitude and behavioural implications and can impact negatively on the ability of a meeting to reach its goal and support of future meetings. Perception of meetings can also have an effect on employee morale. In 2012 there was an apparent lack of people participating in discussions in the plenary assembly meetings. A common theme throughout the three meetings was lack of time for discussions and participants wishing the opportunity to prepare themselves in greater detail before the meeting. Statistically significant benefits have been found in the academic literature with prior access to an agenda and it is believed that this helps in setting expectations in advance of the meeting and allows for pre-meeting preparation. 61 The working environment of meetings is very important and emphasises the finding that this affects mood and influences behaviour. 61 To make meetings more interesting the format of presentations in the plenary assembly meetings can be changed, for example using group breakout sessions and different visual presentations (e.g. flip charts, video clips, etc.). Other important facets of a meeting include a productive, morale-boosting nature and presentations of adequate content and length. 61

Meeting attributes

In 2011 and 2012 participants were asked to rate attributes of the meeting in a combined closed and open question.

The factor that was rated very good most often was the social event. The second best was meeting or networking with colleagues, with 97% of participants in 2011 and 99% in 2012 rating this as good or very good. An important facet of building trust within an international team is providing informal interaction opportunities. 58 Establishing trust is integral for a successful international project, but is difficult to establish. It cannot be achieved by a one-off meeting but takes time and is a reiterative process in order to cement relationships. 58 The importance of combining a team meeting with a team-building activity, such as having a meal in a restaurant, has been emphasised. 58 Reviews of research have highlighted the importance of initial face-to-face meetings in ‘jump starting’ effective virtual teams. 59

It was noticeable that the meeting venue in 2011 was poor – only 8% rated this as very good compared with 64% in 2012. Leadership of the meeting was also better in 2012 than 2011 with 95% rating this as good or very good in 2012. In decision-making meetings the chairperson should use a participatory style rather than an autocratic one. At the end of a meeting the discussions should be summarised and actions denoted. 62

Views of project participants about the involvement of stakeholders

Participants were asked open questions in 2011 and 2012 about concerns associated with involvement of stakeholders in the project. They were asked what the benefits of stakeholder involvement had been by an open question included in the 2011 questionnaire.

When participants were asked in 2012, only two-fifths had no concerns about the actual involvement of stakeholders in the EUnetHTA JA. A further two-fifths did not know if they had concerns and one-fifth had concerns. Only one-third of participants had no concerns about the level of commitment of stakeholders, one-fifth had concerns and half did not know. The qualitative data showed that concerns centred around lack of participation and commitment of stakeholders and the heterogeneous nature of stakeholders. Therefore, there was a large proportion of concern or apparent lack of knowledge, which might have indicated a lack of added value.

Response rates to stakeholder questionnaires

For the S2011 survey, only half of the four industry representatives and half of the four patients/consumers responded. The situation was repeated in S2012 (although the individual non-responding organisations were different). It may be that lack of response is itself of value (e.g. leading to the speculated conclusion that the stakeholder organisation does not consider EUnetHTA of sufficient importance to engage with), but it would be preferred to have this opinion explicitly stated. Unfortunately, it was beyond the scope of the evaluation (agreed with the EU Commission) to include follow-up of non-respondents.

Formation of the stakeholder forum

A combination of closed and open questions were included in the questionnaires about the formation of the forum, its purpose, why stakeholders had applied to join it, what they could contribute and whether or not appropriate organisations had been included.

A prerequisite of the EUnetHTA JA project was an acknowledgement of the importance of communicating with stakeholders. 13 With this aim a formal stakeholder forum was established in 2010. This contained four seats for stakeholder categories of industry, patients/consumers, providers and payers. Stakeholders became aware of this forum from various avenues, including from the EU Commission, the EUnetHTA secretariat and key project personnel and other stakeholders. Stakeholders had applied to be an external stakeholder for various reasons, including contributing to the JA project, to enable reciprocal working patterns and ensuring a balance of stakeholder expertise. Stakeholders believed they could contribute to the project by providing specialist HTA knowledge, providing access to experts, providing a governance role and disseminating the project results. They saw the purpose of the forum as facilitating the stakeholder involvement process by providing an intermediary link between the stakeholders and the JA project, overseeing work package and SAG activities, developing consensus for decision-making and being ‘transparent, responsive, accountable and participative’.

In practice, only two organisations applied for the four seats in both the provider and payer categories. This only represents half-capacity take-up and it is suggested that possibly these were inappropriate types of stakeholder expertise to include in the project or that the call for stakeholders was too limited. There were more participants than spaces for the industry and patients/consumer groups, and this meant that some applicants were excluded. The perceived risk from non-inclusion of these organisations were suggested as missing expertise, decreased scope of JA activities, decreased confidence in the project and inclusion of less experienced representatives. It was notable that only one-fifth thought that the way of setting up the forum was straightforward. The qualitative data showed that concerns included the process itself, that it was organised to fulfil a certain type of stakeholder involvement, restricting stakeholder members prevented equal contributions, lack of clarity about why applicants were rejected and complexity in assigning a representative. However, about three-quarters of stakeholders questioned near the end of the project thought the formation of the forum had been an effective mechanism for involving stakeholders. Over two-thirds of respondents surveyed in 2012 thought the appropriate organisations had been included. Those members who had applied to join the forum but had been excluded from the forum were represented by a representative for their specific group. At the start of the project the majority of these organisations were concerned that they would not be represented adequately. However, these concerns appeared unfounded because by the end of the project all agreed that they had been kept adequately updated by either their forum representative or by the secretariat.

The qualitative data showed that there were apparent initial concerns about the formation of the forum because it was seen as a complicated business and there were concerns about restriction of expertise. However, by the end of the project this was seen as having been an effective mechanism and mostly included the correct stakeholders. Those who had not been successful in gaining a place had been kept informed by their representative.

Operation of the stakeholder forum

A combination of closed and open questions were included in the questionnaires to identify stakeholders’ views about the operation of the forum, whether or not it was fulfilling its purpose and comments on the meetings held.

In 2011, all respondents agreed that the forum was fulfilling its purpose, recognising an initial start-up period. It was seen as being inclusive, evolving and that there was increasing trust. Its purpose was cited as being a special advisory group to the executive committee, commentating on work package tasks, overseeing SAGs, gaining consensus for decisions. The majority thought membership was involving what they had envisaged and it seemed to be improving over time. However, in 2012 about one-third thought it was not fulfilling its purpose and about one-third did not know if it was or not. Of the third who thought it was not fulfilling its purpose, half were patient organisations and half were industry. It could be worth probing stakeholders in greater depth to analyse why the functioning of the forum appeared to depreciate between 2011 and 2012 (but unfortunately this was beyond the scope of the evaluation agreed with the EU Commission).

Regular stakeholder meetings were held (both electronically and face-to-face) and almost three-quarters thought these had been useful. In general, these seemed to be appreciated, were well organised and of a suitable frequency. Suggestions for improvement included having pre-preparation materials sent in advance, a stakeholder chairperson, facilitating more contribution from stakeholders and more suitable agenda items following greater dialogue about setting the agenda.

Half of respondents had concerns about the principles of stakeholder involvement in the EUnetHTA JA (stakeholder policy and standard operating procedures). Of these five organisations all types of stakeholder were included.

Nature of being a stakeholder of the European Network for Health Technology Assessment Joint Action

A combination of closed and open questions were included in the questionnaires about what it meant to be a stakeholder in the project: how they had been involved, how their opinions and expertise had been used, if adequate feedback had been received and how involvement of stakeholders could be improved in EUnetHTA JA2.

At the start of the project stakeholders thought their involvement and contributions would include general input and specific HTA knowledge/expertise, attending meetings, being involved in work packages, facilitating experts, sharing information, providing a governance role and disseminating project results. In the interim year, almost three-quarters of stakeholders indicated that involvement in the stakeholder forum had involved what they thought it would, although some concerns centred around balance of the forum’s representativeness. During the project and at the end, they commented how they had been involved. This broadly included the activities they had anticipated, along with being a member of SAGs, participation in the plenary assembly, organising a joint industry response, regular contact with work package leaders, participating in the EUnetHTA JA conference and in discussions about the future of EUnetHTA JA2. However, it was of concern that almost three-quarters had worries about the actual involvement of stakeholders in the EUnetHTA JA1. Half did not think that their organisation’s expertise had been appropriately used. Of these five organisations, two were from industry, two were patient organisations and one was a payer organisation.

Half had concerns about the ‘principles of stakeholder involvement in the EUnetHTA JA’. The majority of stakeholders either had concerns about the commitment of stakeholders in the JA or did not know if they had concerns or not. Of the four who had concerns, two were from patient organisations, one was from industry and one was from providers.

About one-third of stakeholders thought that stakeholders’ views were not adequately considered and the proportion of those who thought that were considered increased from one-third to one-half by the final year. The two organisations which disagreed were industry and patient organisations. Suggestions for improvement included involving stakeholders at an earlier stage in iterative planning, producing summaries of contributions and making the forum more participatory in nature.

In the interim year, the majority thought that being a member of the stakeholder forum had been a good use of their organisation’s time, but this decreased to two-thirds in 2012. The organisations which thought it had not been a good use of time were both patient organisations. In contrast all organisations who were not participants of the forum had ‘got what it hoped by being a Stakeholder of the JA’ and that it had been a useful use of their time. Participation was seen to confer benefits to stakeholders; increasing general HTA knowledge, recognition for the umbrella stakeholder organisation and ability to comment on documents before publication.

Stakeholder advisory groups were formed for WP4, WP5 and WP7 with the purpose of organising stakeholder input into the work of the EUnetHTA JA. This way of working was apparently well received, with about two-thirds of respondents appreciating this style of involvement. However, the qualitative data showed that concerns included short timelines for responding to consultations and the diverse nature of the stakeholder forum made it difficult to balance points of view.

The stakeholders did not know at baseline whether or not they would be provided with adequate feedback about the project and they suggested that a stakeholder chairperson be appointed to better link them to the project. However, these fears seemed to be unfounded because the majority thought they had received adequate feedback in the interim and final years. There were still some suggestions for improvement, such as that having earlier information about work packages would enable better participation. The one organisation that disagreed was a patient organisation.

There was some apparent concern by some members that the forum was a homogeneous group of experts who could be biased (particularly towards the pharmaceutical industry). In the future, efforts must be made to ensure greater participation and representation of all groups’ views.

Improvement in European network for Health Technology Assessment Joint Action 2

Stakeholders considered that positive involvement had been seen through SAGs, that the diversity of different stakeholder groups on the forum needs to be managed and that earlier involvement of stakeholders, such as in formulating the project objectives and evaluation criteria, would be appreciated. Participants also thought that stakeholders should be involved earlier in the process, that the involvement of stakeholders should prevent bias towards industry, that the forum had been a good mechanism and that communication between project participants and stakeholders should be improved.

Purpose

The majority of stakeholders, and almost three-quarters of participants, thought the EUnetHTA JA2 would serve as a useful follow-up to the EUnetHTA JA project, with the rest not knowing whether or not it would be. Comments revolved around the fact that EUnetHTA JA2 would benefit from the experiences of the EUnetHTA JA, that it is a necessary follow-up project, that it has a different focus, that further development of tools for HTA is required, there is a need to test the effectiveness of joint report production, this would identify co-operation between agencies and enable examination of the methods developed in actual practice. It was seen as a bridge between the EUnetHTA JA and the permanent network. This seemed to be an indication that the EUnetHTA JA had not completely fulfilled its objectives and that it needed a further joint action project to be able to actually apply the tools in practice.

Concerns

The proportion of project participants who had concerns about the EUnetHTA JA2 project decreased from 2011, when over half had concerns, to 2012, when approximately one-fifth had concerns. Different concerns were noted in the 2 years. The qualitative data showed that concerns in 2010 included worries about an overlap with the current JA, HTA reports, poor definition of the project and political aspects. The qualitative data showed that concerns in 2011 included integration of deliverables, scope of the project and involvement of the correct expertise. Lack of resources was noted in both years. Almost one-third of stakeholders had concerns, which included stakeholders not being sufficiently involved or representation of different stakeholders. These also included commitment of governments to implement outputs and the need to have clear obligations and a timetable. About one-third of project participants had concerns about the actual planning process, which included lack of time in submitting the plans, and problems with the overlap between the two projects. About three-fifths thought it was having an impact on the time available for work on the EUnetHTA JA project. Suggestions for how it could have been better included having more time for preparatory work, a bigger focus on capacity building, greater stakeholder input and incorporating recognised project management techniques.

Lessons learned from the European network for Health Technology Assessment Joint Action

Project participants and stakeholders had different perspectives on what could be learnt from the EUnetHTA JA to inform the next project. Participants recognised the importance of collaboration and communication and contribution of participants, better project planning and budgeting, more accurate scoping of work at the start, information sharing, development of tools and appropriate involvement of stakeholders and experts. Stakeholders felt less complexity and greater transparency was needed, a formalised lessons learned document was required from the EUnetHTA JA to build on and stakeholder involvement should be strengthened.

Involvement of stakeholders

Suggestions for improving the involvement of stakeholders included earlier involvement, need for more balanced representation to avoid industry bias, recognition that they have an important role in validating the tools, sharing deliverables with them, the positive effect of SAGs and to share timelines earlier. This indicates the impression that stakeholder involvement was a developing process; it was recognised as needed from the EUnetHTA 2006–8 project, developed through the EUnetHTA JA project with the establishment of the stakeholder forum and SAGs and needs further evolvement in the EUnetHTA JA2 project. In this respect, it is hoped that the lessons learned from the evaluation of the JA project will be helpful.

Secure funding and maintain a dedicated co-ordinating secretariat

The EUnetHTA 2006–8 project was cofunded by an EU grant and the individual participating organisations in a ratio of 1 : 1. For the EUnetHTA JA this was also part funded, in a 1 : 1 ratio from the EU and from Associate partner organisations. 13 As for the EUnetHTA 2006–8 project, the main partner was the Danish Health and Medicines Authority (formerly the National Board of Health). It acted as a dedicated co-ordinating secretariat and led the co-ordination work package (WP1). One aim of this work package was to ensure the submission of technical reports to the EU Commission on time and meeting deadlines of deliverables.

In the EUnetHTA JA project, the co-ordinating secretariat was maintained. Evaluation results showed that approximately three-fifths of project participants thought they had provided effective administration and three-quarters thought their communication by e-mails was adequate. Other activities that could be performed by the secretariat included a greater role in facilitating relationships, providing project management guidance and providing greater feedback about the project. It is suggested that this be considered for EUnetHTA JA2.

Conclusion: Funding was secured for the JA project. The secretariat was maintained as a dedicated co-ordinating support.

Assure efficiency through an organisational structure made up of work packages managed by a core of dedicated partners, with less committed partners taking part as a wider review group

The evaluation of the 2006–8 project concluded that ‘work packages have proven to be a good working model’, while recognising that clarification was needed about how each partner could contribute. The structure of work packages had been shown to be effective by the previous project and is a common method of organising the structure of international projects. 58 Therefore, this structure was chosen again for organisation of the EUnetHTA JA project.

The majority of participants and stakeholders who expressed an opinion in the evaluation of the EUnetHTA JA thought this method of organising the work was positive. However, approximately one-quarter of participants thought there would also be negative aspects associated with this division. Concerns centred around possible overlap between the aims and work of individual work packages and the importance of effective communication between them.

Each work package was led by a designated lead partner organisation and the executive committee brought these organisations together, along with the plenary assembly chairperson and EU representative, every 2 months.

In the EUnetHTA JA project there were some apparent suggested cases of unequal contributions from partners. Going forward to the EUnetHTA JA2 project there are plans to manage contributions by classifying organisations as either ‘active’ or ‘less active’ and managing financial reimbursement accordingly.

Conclusion: The same work package format (common to DG SANCO-funded projects) was used for the EUnetHTA JA. Some concerns about apparent unequal contributions means that organisations will be graded for the level of their activity in the EUnetHTA JA2.

Continue developing and evaluating the tools as necessary

Evaluation of the previous project identified that tools had been produced within the project lifecycle according to the scheduled timeframe. However, it identified that adjustments and further development of ideas was needed and, therefore, tools had not been piloted in a real-life setting (although two pilots had been performed for the HTA Core Model; ‘core HTA on drug-eluting stents’ and ‘core HTA on multislice computerised tomography angiography’). One of the three specific objectives of the EUnetHTA JA project was ‘application and field-testing of developed tools and methods’.

The documentary analysis revealed that most of the 11 project deliverables had been produced by the end of the EUnetHTA JA project. These included the following tools; HTA Core Model on screening, an ‘operational web-based toolkit including database-containing information on evidence generation on new technologies’ and a ‘quarterly communication protocol for information flow on ongoing/planned national assessments of same technologies’. It was predicted that the ‘methodological guidance that will be appropriate for the assessment of relative effectiveness of pharmaceuticals’ would be delivered after the project end, in February 2013.

Although the HTA tools had been further tested in the EUnetHTA JA project there seemed little opportunity to evaluate their use in practice. Therefore, one of the main tasks of EUnetHTA JA2 must be to pilot the tools in actual practice of preparing HTAs.

Conclusion: The HTA tools and methods had been further developed in the EUnetHTA JA project. However, there had been little opportunity to evaluate their use in practice and this appears to be the task of the EUnetHTA JA2 project.

Involve people in the work to ensure commitment, a high level of knowledge and a broad basis for decision-making processes

One of the apparent added values of the EUnetHTA JA project was the opportunity for project participants to network and share information with each other. This contributed to achieving a high level of knowledge within the project.

Some considerable turnover in project participants was seen during the lifecycle of the project and it is important to develop a suite of induction materials to involve new staff in the work quickly and efficiently. In view of this turnover it was of concern that only two-fifths of respondents were aware that their organisation had a sufficient plan to mitigate this in succession planning. Commitment to the work might be affected by lack of staff or resources and careful budgeting and project planning would help this in the EUnetHTA JA2.

Conclusion: One of the added values of participation in the project included information sharing. Commitment to the work was demonstrated by the fact that all deliverables were produced according to the work plan. Potential turnover of staff must be managed and processes implemented to manage resources.

Encourage collaboration and communication among all parties to ensure coherence within groups and within European network for Health Technology Assessment

It is slightly ambiguous what the evaluation considered as ‘all parties’ and ‘groups’. Members seemed to collaborate on the work effectively within the separate constituent work packages. It was notable that one of the main benefits of the EUnetHTA JA project was seen by participants as networking. There seemed to be little evidence of tangible collaboration on HTA during the EUnetHTA JA, although some was performed using the HTA Core Model. More collaboration is planned during the EUnetHTA JA2. Communication within the EUnetHTA JA appeared to be effective and the mechanism most preferred by project participants was face-to-face meetings.

Conclusion: Members appeared to collaborate on the tasks within work packages. There was less evidence on collaboration on HTA reports, but this will be progressed in the EUnetHTA JA2 project. Communication seemed to be effective and face-to-face meetings were the preferred mechanism.

Continue developing the communication platform and clearinghouse functionality to make European Network for Health Technology Assessment the central reference point for HTA in Europe

This was not continued during the JA project.

Conclusion This was not continued in the JA project.

Arrange face-to-face meetings at the start of group or committee work to strengthen social coherence and reach a common understanding of the work

Networking was highly valued by project participants and face-to-face meetings were the preferred means of communication. It has been suggested that such meetings are important in fostering trust between virtual teams of an international project.

Conclusion: Face-to-face meetings seemed highly valued by project participants and should be arranged for the start of group work in the EUnetHTA JA2.

Evaluate the tools in real work settings and the technical communication platform

As discussed previously, the focus of the EUnetHTA JA seemed to be on the development of the tools. The focus of the EUnetHTA JA2 will be on evaluating the tools in real work settings. The technical communication platform of the information management system performed well. The intranet site was used by most members. However, some improvements in the workrooms and for the e-meeting system were required.

Conclusion: There appeared little chance to evaluate the tools in real-life work settings, and this will be the focus of EUnetHTA JA2. The information management system supported the communications of the project well.

English has been the main language and should continue to be so

In an international project it is practical if a common language is used. It was encouraging that the majority of participants had not experienced a serious difficulty with either written or spoken English.

Conclusion: Communicating within the project in English has worked well and this should continue for the EUnetHTA JA2 project.