Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier( 15.04.2008 )
3-page executive summary
Over the past several years there has been a great deal of international discussion about how to harness health research more effectively to achieve the United Nations' Millennium Development Goals (MDGs) as well as other national health goals in low- and middle- income countries (LMICs). In November 2004, country delegations at the Ministerial Summit on Health Research held in Mexico City backed calls for establishing mechanisms to support the use of research evidence in policy and practice, as did the World Health Assembly in May 2005 when it approved a resolution arising from the Mexico Summit. We sought to inform deliberations about next steps by identifying organisations around the world, and especially in LMICs, which are in some way successful or innovative in supporting the use of research evidence in the development of clinical practice guidelines (CPGs), health technology assessments (HTAs), and health policy, and by describing their experiences.
We convened a project reference group, which provided feedback on our approach and materials. We undertook the project in three phases -- 1) a survey, 2) telephone interviews, and 3) case descriptions that drew on site visits -- and in each of the second and third phases we focused on a purposive sample of those involved in the previous phase. We drew on many people and organisations around the world to generate a list of organisations to survey. We modified a questionnaire that had been developed originally by the AGREE collaboration, adapted one version of the questionnaire for organisations producing CPGs and HTAs and another for organisations supporting the use of research evidence in developing health policy (government support units, or GSUs), piloted both versions of the questionnaire, and made a small number of final modifications to both versions of the questionnaire. We sent the questionnaire by email to 176 organisations and followed up periodically with non-responders by email and telephone. We then purposively sampled 25 organisations from among those who responded to the survey. We developed and piloted an interview schedule and conducted interviews by telephone with the director of each organisation. We then purposively sampled eight cases of one or more organisations bridging research and policy from among the cases described in the telephone interviews and (once) other cases with which we were familiar. We developed and piloted a case study data-collection protocol and conducted site visits for each case. Data collection included interviews with 51 key informants and a review of publicly available documents. We conducted simple descriptive statistics using the survey data and we analysed the written survey responses, telephone interviews, in-person interviews, and documents using a constant comparative method of analysis. We produced a video documentary about each case study.
The seven main recommendations that emerged from the advice offered in the telephone interviews provided a remarkably clear way to organise the principal findings and their implications for other organisations.
1. Collaborate with other organisations This advice was reinforced by: 1) the (quantitative) survey finding that more than half of the organisations (and particularly HTA organisations) reported that examples from other countries were helpful in establishing their organisation; 2) the (qualitative) survey finding that many organisations producing CPGs or HTAs conducted a focused review of one particular organisation that they then emulated or a broad review of a variety of organisational models; 3) the (qualitative) survey finding that the advice that was most commonly offered by organisations producing CPGs, HTAs or both was to seek support from similar existing organisations or networks, whether through informal interactions, study tours, mentoring relationships, twinning, partnerships or network memberships; 4) the (qualitative) survey finding that working within national networks and, more generally, collaborating rather than competing with other bodies, was a commonly cited strength in how GSUs are organised; and 5) the case descriptions finding that one of the two types of advice offered to other organisations was to learn from other organisations.
2. Establish strong links with policymakers and involve stakeholders in the work This advice was reinforced by: 1) the (quantitative) survey finding that a high proportion (88%) of GSUs involved target users in the selection of topics or the services undertaken; 2) the telephone interview finding that, while informal relationships with policymakers were identified more frequently as important by GSUs than by organisations producing CPGs, HTAs or both, nearly all of the organisations reported using personal communications with decision-makers, particularly policymakers; 3) the telephone interview finding that organisations both within and outside government viewed their close links with policymakers as a strength; 4) the case descriptions finding that the existence of a strong relationship between researchers and policymakers was repeatedly cited as one of two key organisational strengths; although this strength brought with it a related challenge, namely the need to manage the conflicts of interest that can emerge in any close relationship between researchers and policymakers.
3. Be independent and manage conflicts of interest among those involved in the work This advice was reinforced by: 1) the (qualitative) survey finding that independence is by far the most commonly cited strength in how organisations producing CPGs and HTAs are organised; and 2) the case descriptions finding that the presence of conflicts of interest was repeatedly cited as one of two key organisational weaknesses.
4. Build capacity among those working in the organisation This advice was reinforced by: 1) the quantitative survey finding that most organisations have a small number of full-time equivalent (FTE) staff; 2) and the case descriptions finding that developing capacity among and retaining skilled staff and collaborators was one of their two frequently offered types of advice.
5. Use good methods and be transparent in the work This advice was reinforced by: 1) the (quantitative) survey finding that between 84% and 100% of organisations reported providing panels with or using systematic reviews; 2) the (qualitative) survey finding that an evidence-based approach is the most commonly cited strength of the methods used by organisations that produce CPGs and HTAs; 3) the telephone interview finding that using rigorous methods that are systematic and transparent (sometimes shortened to "being evidence based") was the most commonly cited strength among all organisations; and 4) the case descriptions finding that the use of an evidence-based approach was one of two organisational strengths that were repeatedly cited. However, all but one of the organisations producing CPGs, HTAs or both used informal methods for setting priorities; relatively few organisations producing CPGs and HTAs convened groups to develop CPGs or HTAs, took equity considerations into account or had established a process for addressing conflicts of interest; and GSUs were less likely to have a manual that described the methods they use and to conduct or use systematic reviews and more likely to report using non-systematic methods to review the literature. In addition, using systematic and transparent methods brought with it a related challenge, namely the time-consuming nature of an evidence-based approach.
6. Start small, have a clear audience and scope, and address important questions This finding was reinforced by: 1) the (qualitative) survey finding that the most commonly cited weakness in how these organisations are organised is a lack of resources, both financial and human; 2) the (qualitative) survey finding that the most commonly cited weakness of the methods used by organisations that produce CPGs and HTAs was their time-consuming and labour-intensive nature; 3) the (qualitative) survey finding that GSUs advised others establishing a similar organisation to attend to the need for secure funding; 4) the telephone interview finding that the weakness noted by most of the CPG- and HTA-producing organisations was inadequate resources, more specifically insufficient numbers of skilled staff and time, together with using labour- and timeintensive processes that limit the number and quality of CPGs and HTAs that can be produced and updated; and 5) the case descriptions finding that a lack of resources was repeatedly cited as one of two organisational weaknesses.
7. Be attentive to implementation considerations even if implementation is not a remit This advice is reinforced by: 1) the (quantitative) survey finding that less than half of all organisations provided a summary of take-home messages in their products; 2) the (quantitative) survey finding that between one half and two thirds of organisations do not collect data systematically about uptake; 3) the (qualitative) survey finding that the most commonly cited weaknesses of CPG- and HTA-producing organisations' outputs are the lack of dissemination and implementation strategies for the outputs and the lack of monitoring and evaluation of impact; 4) the telephone interview finding that most organisations argued that it is the clients who requested a CPG or HTA, the minister of health or more generally the department of health who is responsible for implementing recommendations or policy decisions; 5) the telephone interview finding that all types of organisations tended to focus largely on weaknesses in implementation when asked about both strengths and weaknesses, with few exceptions; and 6) the telephone interview finding that most of the examples of success among organisations producing CPGs, HTAs or both were occasions where there was a perception that clinicians adhered to the organisation's recommendations or policymakers based their decisions (at least in part) on the work of the organisation.
The study has six main strengths: 1) we examined the views and experiences of those familiar with three types of organisations that support evidence-informed policymaking, not just one of the two types of organisations previously studied (i.e., we surveyed GSUs as well as CPG- and HTA-producing organisations, we interviewed roughly equal numbers of CPG- and HTA-producing organisations and GSUs, and the majority of case descriptions were GSUs); 2) we achieved both breadth (through a survey) and depth (through telephone interviews with directors and then case descriptions that drew both on interviews with a range of staff, advocates and critics and on documentary analyses) in our examination of their views and experiences; 3) we drew on a regionally diverse project reference group to ensure that our draft protocol, study population, questionnaire, interview schedule, and case study data-collection protocol were fit for purpose; 4) we adapted a widely used questionnaire and achieved a high response rate with our survey (86%); 5) we used explicit sampling criteria to identify particularly successful or innovative groups for more in-depth study through telephone interviews and case descriptions, no organisation declined to participate in the telephone interviews, and only one individual declined to participate in the interviews conducted as part of the site visits; and 6) we employed a variety of independent checks on the credibility of our thematic analyses of the written questionnaire responses and the telephone interview and case study data. The study has two main weaknesses: 1) despite significant efforts to identify organisations in low- and middle-income countries, just over half (54%) of the organisations we surveyed and just under half (48%) of the organisations we interviewed were drawn from highincome countries; and 2) despite efforts to ask questions in neutral ways, many organisations may have been motivated by a desire to tell us what they thought we wanted to hear (i.e., there may be a social desirability bias in their responses).
Participants regard an evidence-based approach as the greatest strength in the way these organisations conduct their work. They see the time-consuming nature of an evidencebased approach as the greatest weakness. They view relationships between researchers and policymakers as highly desirable, but there appears to be little awareness of the nature of potential tensions that can arise and how to manage or resolve them. A lack of resources, both financial and human, poses a challenge in many organisations. Conflicts of interest are seen as a critical issue. Multi-disciplinary teams and international networks are seen as highly desirable, and there is a strong perceived need for coordination at an international level to avoid duplication of processes. Little effort is put into dissemination and implementation activities in relationship to the efforts that are focused on producing evidence-based materials. Negligible efforts are put into communicating evidence to the wider public, via the mass media, and beyond stakeholder constituencies.
1-page key messages
>3-page executive summary