Methods

TRAINING THE INFORMATICIST

The informaticist was a vocationally trained GP interested in evidence-based medicine but with no previous experience of providing a service of this type. Her training involved a series of eight 90-minute tutorials with a senior medical librarian during which she learned how to do rigorous searches of important medical databases. She also attended the 4th London Workshop on Teaching Evidence-Based Health Care and had three tutorials on critical appraisal with MP, a senior lecturer in primary care. Her learning was supplemented by background reading of textbooks on evidence-based medicine.^17,18

The informaticist continued to work half-time as a GP for the duration of the project.

RECRUITMENT OF PARTICIPANTS

All 100 GPs and nurse practitioners (“clinicians”) of the Fulham and Hammersmith primary care groups (box 1) were offered the service free of charge. Two mailings were sent and interested clinicians were offered a practice visit by one of the investigators. Thirty four clinicians (32 GPs and two nurse practitioners) registered from 14 practices.

DELIVERY OF THE SERVICE

Participants submitted questions on a short request form adapted from a form developed by Hayward et al.¹⁴ If additional information was required to formulate the question, this was discussed with the questioner (n=8).

The search cascade shown in box 2 was used, starting with a search of pre-existing databases of evidence. If questions could be answered fully at one level, lower levels of the cascade were not searched routinely. A primary literature search (step 8) was conducted if insufficient evidence was found earlier. Some questions could be answered partially at one level in the cascade, but also required use of resources further down the cascade. Medline was used for primary literature searching as there are well developed “quality filters” for this database. Unpublished evidence was not sought and, when good evidence was unavailable, this was made explicit.

The informaticist critically appraised the literature using recognised critical appraisal checklists and returned an evidence-based summary to the questioner. A standard format was used, a modification of the CAT (“critically appraised topic”),¹⁸ consisting of:

• The Question (asked by the questioner)

• The Modified Question (if question reformulated)

• The “Clinical Bottom Line” (the main message, in bold type)

• The Evidence (summary of the critical appraisal)

• Reference list

Participants were invited to request original references, although none did. A database of questions and answers was available to participants on the project website¹⁹ and in project newsletters.

Results

THE QUESTIONS

Twenty two of the 34 participants used the service (20 GPs and two nurse practitioners), generating 60 questions of which 57 could be answered. Examples of questions asked are shown in box 3. Twelve GPs failed to use the service, seven asked one question each, while the two highest users asked 13 of the 60 questions (22%). Fourteen of the 60 questions originated from a single group practice.

The classification and origin of the questions is shown in table 1 with information on how the questions would otherwise have been pursued.

THE ANSWERS

Answering the questions was time consuming, taking a median of 130 minutes (range 25–450). Median turnaround time, defined as the time between the date on the request form and the answer date, was 9 days (range 0–34); 72% of questions were answered within two weeks. Table 2 shows the distribution of resources used in answering questions.

EVALUATION OF THE SERVICE

A questionnaire was sent with each evidence-based summary to assess satisfaction with the answers provided and to find out how the information was used. A 95% response rate was achieved, details of which are shown in table 3.

At the end of the 10 month period 17 (77%) of the 22 participants who had asked questions agreed to participate in a final evaluation of the service based on a semi-structured questionnaire. Of these, 12 were interviewed (two face to face, 10 by telephone) and five submitted the completed questionnaire by fax. We explored the participants' overall impressions of the service, how they had used the information, how their behaviour had changed as a result of the service, and how they saw the service fitting into general practice.

The comments added further support to the positive evaluation derived from the individual questionnaires. High levels of satisfaction were expressed and most commented on the expertise of the informaticist. Six said they would have like to make more use of the service, reporting “time” and “lack of organisation” (in the practice) as barriers. Eleven respondents said that access to the service had increased their likelihood of asking questions and 10 said it had increased their likelihood of pursuing answers. Seven respondents said it had increased their likelihood of searching for answers themselves. Most were enthusiastic about the potential role for this kind of service in the future of general practice, although for some this was tempered with concerns about the potential costs involved. Some direct quotes from this part of the evaluation are shown in box 4.

Discussion

We have shown that a clinical informaticist service was a valuable method of providing relevant, high quality information to primary health care workers. Satisfaction with the service by those who used it was very high, in line with the suggestion that doctors value summarised evidence.^7,8 Users reported that changes in practice would occur, and reported other positive spin-offs such as increased understanding and promotion of discussion with colleagues. Although it is a small study based on 34 interested participants in a small geographical area, we have collected useful data on the processes involved in providing such a service, including the types of resources accessed in the answering process. Despite a growing body of systematic reviews and evidence databases, primary research literature was accessed to answer 61% of questions. This was similar to the findings of Hayward et al who used primary literature to answer 58% of questions.¹⁴ Rigorous critical appraisal of primary literature is a time consuming process, requiring access to full text articles and frequent requests for interlibrary loans. This (and the fact the informaticist was part time) accounted for the longer than desirable turnaround time (median 9 days). However, although slower than the recently reported ATTRACT service,¹³ satisfaction with the timescales was high and requested timescales were met for 82% of the questions. We also collected more data on the process involved in providing the service and obtained a more thorough ongoing evaluation by requesting feedback on every question handled.

Appropriate implementation of evidence requires incorporation of the information in the individual clinical context. This developmental study was not designed to assess objective measures of change, relying solely on outcomes reported by users of the service. Further research is needed to establish whether providing an informaticist service results in better health outcomes, and how this strategy compares with other possible methods of increasing access to evidence.

Uptake of the service was lower than anticipated. There are various reasons why uptake may have been poor and this should be the focus of further research. It may be related to the time required to submit questions, although the service was designed to ensure this was minimal. It takes time for clinicians to incorporate novel services into clinical routines; 10 months may have been insufficient. Some GPs may not perceive the evidence-based medicine approach as helpful to their day to day work.²⁰

There is still much to be learned about the best search strategy to employ. The cascade we used was systematic and a practical way of using different resources. The order in which the resources were used (especially at steps 2–7 of box 2) does not necessarily equate with “quality” of evidence retrieved at each level and, as such, this should not be regarded as a search “hierarchy”. Neither is it fully comprehensive. However, it allowed a consistent logical approach. Step 7 (other evidence-based medicine web sites) was usually carried out only when the informaticist had prior awareness of a relevant website for a particular question. It is unknown whether using a longer (or, indeed, shorter) cascade would significantly alter the search results or the answers provided. It has been suggested that a pragmatic approach to searching is effective,²¹ but we are unaware of any research on the relative benefits of different approaches in the context of providing evidence-based information for clinicians. It is possible that a flexible approach could be adopted in which users of informaticist services could determine their preferred approach in advance. It is likely that the “stakes” attached to different questions vary, and it may be that different search strategies could be used in different circumstances.

The clinical informaticist service seems to represent a useful additional method of acquiring evidence-based information in primary care. It has the added advantage that the informaticist responds directly to the specific information needs of clinicians and that communication is from clinician to clinician. There are still aspects that require further research, but we have shown that a service like this is highly valued by users and leads to reported changes in practice.