You are here

Article Text

Article menu

Download PDFPDF

XML
Health services research
Research
First quality score for referral letters in gastroenterology—a validation study
  1. Sigrun Losada Eskeland1,
  2. Cathrine Brunborg2,
  3. Birgitte Seip3,
  4. Kristine Wiencke4,
  5. Øistein Hovde5,6,
  6. Tanja Owen7,
  7. Erik Skogestad8,
  8. Gert Huppertz-Hauss9,
  9. Fred-Arne Halvorsen10,
  10. Kjetil Garborg11,12,
  11. Lars Aabakken13,
  12. Thomas de Lange4,14
  1. 1Department of Medical Research, Vestre Viken Hospital Trust Bærum, Drammen, Norway
  2. 2Research Support Services, Oslo Centre for Biostatistics and Epidemiology, Oslo University Hospital, Oslo, Norway
  3. 3Department of Medicine, Vestfold Hospital Trust Tønsberg, Tønsberg, Norway
  4. 4Department of Medicine, Vestre Viken Hospital Trust Bærum, Drammen, Norway
  5. 5Department of Gastroenterology, Innlandet Hospital Trust Gjøvik, Gjøvik, Norway
  6. 6Institute of Clinical Medicine, University of Oslo, Oslo, Norway
  7. 7Department of Gastroenterology, Østfold Hospital Kalnes, Kalnes, Norway
  8. 8Department of Medicine, Innlandet Hospital Trust Lillehammer, Lillehammer, Norway
  9. 9Department of Gastroenterology, Telemark Hospital Trust Skien, Skien, Norway
  10. 10Department of Medicine, Vestre Viken Hospital Trust Drammen, Drammen, Norway
  11. 11Department of Transplantation Medicine, Oslo University Hospital, Oslo, Norway
  12. 12Department of Health Economics and Health Management, University of Oslo, Oslo, Norway
  13. 13Department of Gastroenterology, Oslo University Hospital, Oslo, Norway
  14. 14Cancer Registry of Norway, Oslo University Hospital, Oslo, Norway
  1. Correspondence to Sigrun Losada Eskeland; sigesk{at}vestreviken.no

Abstract

Objective To create and validate an objective and reliable score to assess referral quality in gastroenterology.

Design An observational multicentre study.

Setting and participants 25 gastroenterologists participated in selecting variables for a Thirty Point Score (TPS) for quality assessment of referrals to gastroenterology specialist healthcare for 9 common indications. From May to September 2014, 7 hospitals from the South-Eastern Norway Regional Health Authority participated in collecting and scoring 327 referrals to a gastroenterologist.

Main outcome measure Correlation between the TPS and a visual analogue scale (VAS) for referral quality.

Results The 327 referrals had an average TPS of 13.2 (range 1–25) and an average VAS of 4.7 (range 0.2–9.5). The reliability of the score was excellent, with an intra-rater intraclass correlation coefficient (ICC) of 0.87 and inter-rater ICC of 0.91. The overall correlation between the TPS and the VAS was moderate (r=0.42), and ranged from fair to substantial for the various indications. Mean agreement was good (ICC=0.47, 95% CI (0.34 to 0.57)), ranging from poor to good.

Conclusions The TPS is reliable, objective and shows good agreement with the subjective VAS. The score may be a useful tool for assessing referral quality in gastroenterology, particularly important when evaluating the effect of interventions to improve referral quality.

  • PRIMARY CARE

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2016-012835

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Multicentre design gives better external validity.

    • Wide variety of gastrointestinal indications encompassed by score.

    • Score does not assess appropriateness of the indication for the referral.

    • Potential interindividual differences in the use of the visual analogue scale.

    Introduction

    General practitioners (GPs) refer numerous patients to specialised healthcare services every year, and the referral rates are increasing worldwide.1 ,2 A recent study revealed a referral rate of 13.7% with great variations between GPs.3 The referral letter is a key document for the communication between the GPs and the hospital consultants,4 ,5 and its content and quality are essential for the scheduling and prioritisation of patients. Incomplete, erroneous or extensive referral letters may delay the admission of patients to secondary healthcare services, which may result in delayed diagnosis and a poorer prognosis. It is well documented that referral letters of low quality are prevalent.5–21

    In Norway, referral letters are sent at the GP’s discretion and generated by using free text in a standard template created by the Norwegian health authorities22 (table 1). The template includes the urgency of the referral, but does not include symptom-specific or indication-specific criteria. It is very likely that specific symptoms and clinical findings are crucial for deciding the urgency of the referral.23 The GPs optionally add relevant information about supplementary workup, like laboratory tests and imaging results, by copying and pasting these results to the referral text. Subsequently, the referral is transferred electronically by a secure system to the hospital.24 At arrival, a consultant assesses the referral letter and prioritises the urgency of the referral. If the consultant considers the referral indication inappropriate, it may be rejected. However, the legislation prohibits rejection due to poor quality.

    View this table:
    Table 1

    General requirements for referral letters, according to the Norwegian national referral guidelines, including the frequency of reporting in this study

    The Norwegian Prioritization Guideline for gastroenterology (NPGg),25 created by an expert group of gastroenterologists, states nine main indications for referral to gastroenterology services (open access endoscopy and consultations); dyspepsia, dysphagia, chronic abdominal pain, change of bowel habit, diarrhoea, constipation, gastrointestinal bleeding, weight loss and jaundice/elevated liver enzymes. The guidelines aim to cover 75–80% of gastroenterology referrals. Bowel cancer screening and polyp surveillance is not a part of the indications because this is taken care of either in the hospital system or by population-based trials for bowel cancer screening.

    Despite the well-documented quality issues, no validated and objective tool to assess the quality of referral letters to the specialist healthcare services exists.

    An objective, standardised score would probably be beneficial to ease the assessment of referral quality and to evaluate interventions intending to improve the quality of referral letters.

    The aim of the study was to develop an objective, relevant and reliable quality score for the evaluation of referral letters for the most common indications for referral to gastroenterologists, and to validate the score in a clinical setting.

    Materials and methods

    Development of the score

    We used the nine core indications for referral to gastroenterologists specified in the NPGg:25 dyspepsia, dysphagia, chronic abdominal pain, change of bowel habit, diarrhoea, constipation, gastrointestinal bleeding, weight loss and jaundice/elevated liver enzymes.

    For each of these nine indications, we created a list of 29–36 relevant medical variables based on UpToDate,26 the Norwegian Electronic Medical Handbook for Doctors (NEL)27 and the NPGg.25 Administrative information including sex and date of birth of the patient is included in referral letters by default, as well as name, health personnel identification number and address of the referring physician, and therefore we did not include this information in the nine lists.

    Between November 2013 and March 2014, we invited 39 gastroenterologists within the South-Eastern Norway Health Region to participate in a web-based study. They recorded demographic information such as sex, age, experience and workplace before they started the survey. For each of the nine indications, we asked the participating gastroenterologists to select the 15 most important variables for assessing and prioritising referrals. The importance of the variables was categorised from three points (most important) to one point (less important) with five variables in each category. The remaining variables were given zero points. We then summarised the points assigned to the individual variables for each indication from all the returned questionnaires, and selected the 15 variables with the highest sum of points to comprise the final variables in the score. The five highest rated variables were classified with three points, the next five with two points and the last five with one point.

    After a period of a minimum 6 weeks, we repeated this process to check for reliability of the values assigned. Only the results from the first round were used in the final score. This resulted in a symptom-specific Thirty Point Score (TPS).

    Validation of the score

    Between May and September 2014, seven primary gastroenterology referral centres in South-Eastern Norway Health Region collected 327 referral letters, 21–50 for each of the nine indications stated in the NPGg.25 Patients were mostly referred for open access endoscopy as well as consultations.

    One or two gastroenterologists in each participating centre collected and assessed consecutive referrals within the nine indications. They rated the quality of the referral letter on a 10 cm visual analogue scale (VAS), where 0 cm indicated the worst possible quality and 10 cm the best possible quality. We chose VAS as a comparator to the TPS to assess the external validity in the validation process, assuming that referral letters containing all essential information for assessment and prioritisation of the referred patients would also yield a high VAS.28 The gastroenterologists also recorded whether the referral letters contained too much or too little information, and whether they were unstructured or illegible. Patient age and gender were recorded before all patient data were removed from the referral letters and they were handed over to the study team.

    Subsequently, one researcher from the study team assessed all the referral letters according to the TPS. Both the presence and absence of signs and symptoms were given equal points, as long as they were reported adequately in the referral letter. We also recorded demographic data about the referring physicians (sex, age, county, size of patient list, etc) from publicly available records, and whether the referral letter complied with the national guidelines for referral letters to the specialist healthcare services.22

    Information that was not available to the consultant through the referral letter at the time of assessing the referral, for example, results from laboratory tests and radiology that were communicated later in the referral process, was not included when calculating the TPS.

    To check for intra-rater and inter-rater reliability, we randomly selected 25% of the referral letters from each indication and reassessed them with the TPS. This was done after a minimum of 6 months by the same researcher from the research team and a second independent gastroenterologist. These ratings were done completely blinded from each other and from the results of the first rating.

    Statistics

    We present descriptive statistics as means with their 95% CI or as proportions. We use intraclass correlation coefficient (ICC) for continuous measures and weighted κ statistics for categorical measures to describe reliability of the gastroenterologists' selection of score variables. ICC for average measurements was used to calculate intra-rater and inter-rater reliability of the TPS assessment of referral letters. ICC values were interpreted as: >0.75=excellent, 0.40–0.75=fair to good and <0.40=poor.29 The κ values were interpreted as: >0.80=very good, 0.61–0.80=good, 0.41–0.60=moderate, 0.21–0.40=fair and <0.21=poor.30 Univariable and multivariable linear regression analysis was performed to determine whether patient-related or doctor-related factors were associated with changes in quality of the referral letter, using a manual backward elimination procedure. Any variable with a p<0.25 from the univariable analysis was considered a candidate for the multivariable model. We used the Pearson correlation coefficient to assess the correlation between the VAS and the TPS and ICC for average measurements to assess agreement between the two measurements. We assessed any differences in the VAS/TPS between the different centres by using one-way analysis of variance (ANOVA). The significance level was set at 0.05. All calculations were performed using the IBM SPSS V.21 (IBM SPSS, Chicago, Illinois, USA).

    Power estimation

    Sample size estimation was performed to investigate the association between the VAS and the TPS. The correlation coefficient (r) was anticipated to be 0.60. Considering a probability of a type I error of 5%, 80% power and a two-sided test, at least 19 referral letters were required for each indication. Thus, during the inclusion time, a minimum of 21 referral letters were collected for each indication. Since some indications were more common than others, the number of referral letters for each indication varies upwards.

    Results

    Selection of TPS variables

    Of the 39 invited gastroenterologists, 32 started to record their demographic data, and 26 (81.3%) also moved on to select the variables for the score. Twenty-five (64.1%) of the 39 gastroenterologists completed the whole survey and were included in the study. The excluded gastroenterologist provided answers for one single indication and then aborted the questionnaire. The reason for the dropouts cannot be determined due to the study format.

    The mean age of the included gastroenterologists was 48.5 years (range 35–69). Mean experience as a licensed gastroenterologist was 9.6 years (range 1–33). Sixteen gastroenterologists (64%) repeated the questionnaire 6 weeks later to test for reliability. The characteristics of these gastroenterologists were not significantly different from the nine who did not repeat the survey. The ICC for the reliability of the sum of the scores for the variables was excellent (0.88 to 0.93) for all the indications. The κ values for the reliability of final scores showed a good to very good agreement in all of the indications. The resulting TPS for all nine indications is presented in online supplementary appendix 1. It consists of 15 items for each indication. Depending on the value of the item for the quality of the referral, it is awarded with 1, 2 or 3 points if described adequately in the referral letter. The maximum score for a referral is 30 points, indicating a high-quality referral letter.

    Validation of the score

    The referring physicians were on average 47.1 years old (range 26–72), 37.3% were female and 95% were GPs. The referred patients were 62.4% female, and the average age was 57.2 years (range 6–94).

    Adherence to the Norwegian referral guidelines varied substantially, as shown in table 1.

    In particular, information regarding allergies/critical information, family history of disease and alcohol/tobacco consumption was sparse.

    The average quality of the referral letters assessed by VAS was 4.7 (95% CI 4.5 to 5.0, range 0.2–9.5). The mean TPS for all referral letters was 13.2 (95% CI 12.8 to 13.8, range 1–25). In total, 54.1% of the referral letters had a VAS below 5 (out of 10) and 59.6% had a TPS below 15 (out of 30).

    Intra-rater and inter-rater reliability of the TPS for scoring referral letters was excellent (ICC=0.87 (95% CI 0.81 to 0.92) and 0.91 (95% CI 0.86 to 0.94), respectively).

    The average VAS and TPS for the nine indications are shown in table 2 together with the correlation between the two scores. The VAS and the TPS showed a moderate overall correlation (r=0.42; figure 1), ranging from fair to substantial (r=0.24 to 0.63) for the different indications.

    View this table:
    Table 2

    Referral information quality assessed by VAS and TPS, and correlation between them

    Figure 1
    Figure 1

    Correlation between TPS and VAS

    A multiple linear regression analysis with a manual backward elimination procedure showed that age and gender were the only patient-related or doctor-related variables associated with TPS (age: βadj=−1.156, 95% CI −2.24 to (−0.076)), p=0.036; gender: βadj=−0.090 (95% CI −0.136 to (−0.043), p<0.001), explaining 7% of the variance of TPS (r2=0.07). Further, gender was identified as the only variable associated with VAS (βj=−0.513, 95% CI −0.993 to (−0.033), p=0.036), explaining 1% of the variance of VAS (r2=0.01).

    When the gastroenterologist had recorded that the referral letter contained too little information (n=167 (51.1%)), the VAS and the TPS were also significantly lower (mean difference (Δ)=1.7, p<0.001 and Δ=3.4, p<0.001, respectively). When the gastroenterologists had recorded that the referral letter was unstructured (n=60 (18.3%)), the VAS was significantly lower (Δ=1.6, p<0.001), but the TPS was unaffected (Δ=−0.4, p=0.51).

    There were significant differences in the TPS and the VAS between the centres, and this difference was confirmed by the ANOVA analysis for the TPS (p<0.001) and the VAS (p=0.004). For the TPS, this significant difference disappeared by removing centre III from the calculations. The difference in the VAS disappeared by removing centre II (table 3).

    View this table:
    Table 3

    Relationship between TPS and VAS by centre

    Some analyses were made to identify factors that could increase the correlation between the VAS and the TPS. Eliminating the one-point items did not improve the correlation. Neither did adjusting for the number of three-point items in the referral.

    Discussion

    Overview and principal findings

    This is, to the best of our knowledge, the first study to develop an objective, reliable and validated score (TPS) to assess the quality of gastroenterology referral letters, and it may work as a model for other medical specialties. The score is useful for the majority of referrals in gastroenterology, regardless of the seriousness of the condition or the location of the disease.

    The TPS has demonstrated an excellent intra-rater and inter-rater reliability as well as a moderate correlation between the TPS and a subjective VAS score assigned by gastroenterologists. The quality of the referral letters was variable, both assessed by the TPS and the VAS.

    The correlation and agreement between the TPS and the VAS was somewhat lower than expected (r=0.42, ICC=0.47). Factors not captured by the TPS may also influence the subjective assessment of the quality of referral letters, such as lack of structure or appropriateness. Such factors may have negatively influenced the correlation between the two measurements. Unstructured letters had a lower VAS despite adequate content according to the TPS.

    There were significant differences between the centres both for the mean TPS and the mean VAS. Differences in quality of referral letters in different geographical regions have in other studies been explained by GP workload, referral culture or capacity in local nursing and care institutions, and some of these factors may also be present in this study.23

    The only patient-related or physician-related factor associated with a change in quality of the referral letters in this study was increasing age and male sex of the referring physician, both leading to small decreases in the TPS, but the changes are minor and most likely not clinically relevant.

    The TPS consists of 15 items for each indication. This number may be too high, as many referral letters may contain sufficient information with fewer items.31 However, eliminating the one-point items did not improve the correlation between the VAS and the TPS.

    Also, scoring instruments with many interpretive questions could have a lower reliability.32 Since the TPS demonstrated excellent intra-rater and inter-rater reliability, this does not seem to be an issue with the TPS.

    Some score items that may seem of limited relevance for a given indication (eg, information about Faecal Occult Blood Test (FOBT) in the dyspepsia indication) may have been selected by the gastroenterologists due to the variable's ability to discriminate between serious and less serious diseases.

    In conclusion, the TPS measures the quality of the information in the referral letter objectively but is not a perfect tool for assessment of overall quality of the referral letter, as this also involves consideration of indication and structure of the referral.

    Comparison with the existing literature

    A score to assess referral letters in colorectal surgery has previously been suggested by Jiwa et al.33 This score was developed by colorectal surgeons and GPs, and was validated against the likelihood of detecting colorectal cancer, not against any ‘gold standard’ for referral letter quality or on a wider range of gastrointestinal conditions. Consequently, it is not possible to determine whether high scores actually reflect a high quality of referral letters. Thus, for assessing referral letters in general, the TPS is a more useful tool.

    VAS is a recognised tool for assessing pain,34 and has been shown to be well suited for assessing soft data in clinical trials.35 It has also been used previously in quality score validation studies.28 We therefore chose VAS as a method to assess the overall quality of the referral letters. Others have also used VAS as a tool to assess quality of referral letters,16 and showed scores similar to the findings in our study, with values between 1.1 and 6.9 for the various information items. However, VAS is a subjective measurement, and cannot replace objective scoring methods for evaluating quality.

    Other more general scores for content of referral letters have also been created,7 ,20 but have given little insight into the specific symptom-related items mentioned in the referral letters, and cannot be used to assess the information quality of the referral letter. Also, referral letters in Norway are mainly generated electronically within the general national referral template,36 and these general scores could consequently indicate a good referral letter, regardless of the description of the patient's symptoms and signs.

    Strengths and limitations of the study

    An important strength of the present study is the multicentre design, giving the results a higher external validity. This aspect is also maintained by the wide variety of indications included, covering most of the reasons for referrals to Norwegian gastroenterology units.

    The inclusion of a large number and wide variety of clinical gastroenterologists in the development of the TPS also ensures that the score reflects what the specialists actually need to effectively assess the referral letters. Further, the comparison of the TPS with the gastroenterologists' subjective assessment of the quality of the referral letter (VAS) enhances the emphasis on what the actual assessors value in the referral letters.

    Our study has some potential limitations. First, we have not assessed referral appropriateness and cannot distinguish appropriateness as a deciding factor for lack of correlation between the VAS and the TPS. It is possible that appropriateness of the referral has influenced the VAS, particularly when referrals with poor indication have been well written in terms of clinical information and structure.

    We also observed a difference in the VAS between the centres, and it is possible that systematic differences in the use of the VAS may have influenced the results of the study.

    In this study, gastroenterologists determined the optimal content of referral letters for easy assessment and prioritisation. A score based on risk factors for gastrointestinal cancer could have been an alternative approach. However, the aim was to develop a score that reflects what makes a referral letter easier to assess and prioritise, not to identify high-risk patients.

    Another weakness of the study may be a selection bias for the gastroenterologists who participated in the score development. Thirty-nine gastroenterologists were invited to participate, and the 25 who completed the study may differ from the 14 who did not. However, willingness to participate probably does not influence the validity of their opinions regarding the content of referral letters, and could also be seen as a strength, as an interest in the topic may indicate a better understanding of what should be considered important clinical information in referral letters.

    Some of the items selected for the score may be somewhat too unspecific, or may be considered inappropriate for the indication by other gastroenterologists. We have, for example, chosen the unspecific term ‘previous radiology’ in the jaundice/elevated liver enzymes indication, while ‘previous ultrasound of the liver’ may be a more appropriate and specific item. We have also chosen the term ‘current medical treatment’ as we believe this term accounts for any relevant information regarding the patient's medication, including ingestion of anticoagulants or antiplatelet agents.

    Implications of the study

    We have developed and validated an objective and reliable score for assessing the quality of referral letters in gastroenterology.

    The moderate quality of referral letters observed in this study suggests that a tool to facilitate creation of high-quality referral letters would be beneficial. Information technology, like checklists or clinical decision-making systems, may be a part of the solution.

    Unanswered questions/future research

    This study is a presentation of the TPS that resulted from the survey among the 25 consultant gastroenterologists. However, some refinement of the TPS may be warranted and may increase the ability of the TPS to discriminate between high-quality and low-quality referral letters. Validation of the TPS in other healthcare systems is also necessary to reach a TPS of a high general validity.

    In this study, we found a high prevalence of referral letters with scores below the middle value of the scales for both TPS and VAS. Future research should aim to implement and evaluate interventions to improve quality of referral letters in a way acceptable to referring GPs and specialists in the hospitals. A Cochrane review of interventions to improve referrals identified active involvement of secondary care specialists, and implementation of structured referral sheets as the only interventions with effect on referral quality.37 Electronic referrals are the norm in Norway, and the implementation of structured referral sheets/checklists in electronic referrals may be an interesting intervention to explore.

    Conclusion

    The TPS is reliable, objective and shows good agreement with the subjective VAS. The score may be a useful tool for assessing referral quality in gastroenterology, particularly important when evaluating the effect of interventions to improve referral quality. The method used in the development of the score can serve as a model for other medical specialties.

    Acknowledgments

    The authors specially thank the administrative staff in the local gastroenterology departments for help with collecting paper copies of all included referrals.

    References

    1. ↵
      1. Barnett ML,
      2. Song Z,
      3. Landon BE
      . Trends in physician referrals in the United States, 1999–2009. Arch Intern Med 2012;172:163–70. doi:10.1001/archinternmed.2011.722
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Fleming DM
      . The European study referrals from primary to secondary care. University of Limburg, 1993.
    3. ↵
      1. Ringberg U,
      2. Fleten N,
      3. Deraas TS, et al
      . High referral rates to secondary care by general practitioners in Norway are associated with GPs’ gender and specialist qualifications in family medicine, a study of 4350 consultations. BMC Health Serv Res 2013;13:147. doi:10.1186/1472-6963-13-147
      OpenUrlPubMed
    4. ↵
      1. Long A,
      2. Atkins JB
      . Communications between general practitioners and consultants. BMJ 1974;4:456–9. doi:10.1136/bmj.4.5942.456
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Westerman RF,
      2. Hull FM,
      3. Bezemer PD, et al
      . A study of communication between general practitioners and specialists. Br J Gen Pract 1990;40:445–9.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Gulati S,
      2. Jakola AS,
      3. Solheim O, et al
      . Assessment of referrals to a multidisciplinary outpatient clinic for patients with back pain. J Man Manip Ther 2012;20:23–7. doi:10.1179/2042618611Y.0000000021
      OpenUrlPubMed
    7. ↵
      1. François J
      . Tool to assess the quality of consultation and referral request letters in family medicine. Can Fam Physician 2011;57:574–5.
      OpenUrlFREE Full Text
    8. ↵
      1. Cheng J,
      2. Beltran-Agullo L,
      3. Trope GE, et al
      . Assessment of the quality of glaucoma referral letters based on a survey of glaucoma specialists and a glaucoma guideline. Ophthalmology 2014;121:126–33. doi:10.1016/j.ophtha.2013.08.027
      OpenUrlCrossRefPubMed
    9. ↵
      1. Jiwa M,
      2. Coleman M,
      3. McKinley RK
      . Measuring the quality of referral letters about patients with upper gastrointestinal symptoms. Postgrad Med J 2005;81:467–9. doi:10.1136/pgmj.2004.027516
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Thorsen O,
      2. Hartveit M,
      3. Baerheim A
      . The consultants’ role in the referring process with general practitioners: partners or adjudicators? A qualitative study. BMC Fam Pract 2013;14:153. doi:10.1186/1471-2296-14-153
      OpenUrl
    11. ↵
      1. Ibiyemi O,
      2. Ibiyemi T
      . Quality and contents of referral letters from peripheral health centers to the dental centre of a teaching hospital, southwestern Nigeria. Acta Odontol Scand 2012;70:165–8. doi:10.3109/00016357.2011.600712
      OpenUrlCrossRefPubMed
    12. ↵
      1. DeAngelis AF,
      2. Chambers IG,
      3. Hall GM
      . The accuracy of medical history information in referral letters. Aust Dent J 2010;55:188–92. doi:10.1111/j.1834-7819.2010.01221.x
      OpenUrlCrossRefPubMed
    13. ↵
      1. Hedelin H,
      2. Johansson N,
      3. Strömberg P
      . Uneven quality of referrals for lower urinary tract symptoms in men. Läkartidningen 2003;100:1435–7.
      OpenUrl
    14. ↵
      1. Scully ND,
      2. Chu L,
      3. Siriwardena D, et al
      . The quality of optometrists’ referral letters for glaucoma. Ophthalmic Physiol Opt 2009;29:26–31. doi:10.1111/j.1475-1313.2008.00600.x
      OpenUrlPubMed
    15. ↵
      1. Jack C,
      2. Hazel E,
      3. Bernatsky S
      . Something's missing here: a look at the quality of rheumatology referral letters. Rheumatol Int 2012;32:1083–5. doi:10.1007/s00296-011-1832-z
      OpenUrlCrossRefPubMed
    16. ↵
      1. Garåsen H,
      2. Johnsen R
      . The quality of communication about older patients between hospital physicians and general practitioners: a panel study assessment. BMC Health Serv Res 2007;7:133. doi:10.1186/1472-6963-7-133
      OpenUrlCrossRefPubMed
    17. ↵
      1. Berendsen AJ,
      2. Kuiken A,
      3. Benneker WH, et al
      . How do general practitioners and specialists value their mutual communication? A survey. BMC Health Serv Res 2009;9:143. doi:10.1186/1472-6963-9-143
      OpenUrlCrossRefPubMed
    18. ↵
      1. Jenkins RM
      . Quality of general practitioner referrals to outpatient departments: assessment by specialists and a general practitioner. Br J Gen Pract 1993;43:111–13.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Mead GE,
      2. Cunnington AL,
      3. Faulkner S, et al
      . Can general practitioner referral letters for acute medical admissions be improved? Health Bull (Edinb) 1999;57:257–61.
      OpenUrlPubMed
    20. ↵
      1. Lønning KJ,
      2. Kongshavn T,
      3. Husebye E
      . Kvaliteten på henvisninger fra fastleger til medisinsk poliklinikk. Tidsskr Nor Laegeforen 2009;129(Årg. 129, nr. 18):1868–9. doi:10.4045/tidsskr.09.34967
      OpenUrlPubMed
    21. ↵
      1. Ong SP,
      2. Lim LT,
      3. Barnsley L, et al
      . General practitioners’ referral letters—do they meet the expectations of gastroenterologists and rheumatologists? Aust Fam Physician 2006;35:920–2.
      OpenUrlPubMed
    22. ↵
      Helsedirektoratet. Nasjonal veileder for henvisninger til spesialisthelsetjenesten. Oslo: Helsedirektoratet, 2015.
    23. ↵
      1. Martinussen PE
      . Referral quality and the cooperation between hospital physicians and general practice: the role of physician and primary care factors. Scand J Public Health 2013;41:874–82. doi:10.1177/1403494813498951
      OpenUrlAbstract/FREE Full Text
    24. ↵
      Health-Net N. Norwegian Health-Net. Secondary Norwegian Health-Net. https://www.nhn.no/Sider/default.aspx
    25. ↵
      1. Nygaard E,
      2. Kårikstad V
      . National Prioritization Guideline for gastroenterology. In: Health TNDo, ed. Oslo: The Norwegian Directory of Health, 2009.
    26. ↵
      UpToDate. Secondary UpToDate 2013. http://www.uptodate.com/home
    27. ↵
      Norwegian Electronic Medical Handbook for Doctors. Secondary Norwegian Electronic Medical Handbook for Doctors 2013. http://legehandboka.no/
    28. ↵
      1. Boyer L,
      2. Baumstarck-Barrau K,
      3. Belzeaux R, et al
      . Validation of a professionals’ satisfaction questionnaire with electronic medical records (PSQ-EMR) in psychiatry. Eur Psychiatry 2011;26:78–84. doi:10.1016/j.eurpsy.2009.10.007
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Fleiss JL
      . Design and analysis of clinical experiments. John Wiley & Sons, 2011.
    30. ↵
      1. Altman D
      . Practical Statistics for Medical Research. London: Chapman and Hall, 1991.
    31. ↵
      1. Wåhlberg H,
      2. Valle PC,
      3. Malm S, et al
      . Impact of referral templates on the quality of referrals from primary to secondary care: a cluster randomised trial. BMC Health Serv Res 2015;15:353. doi:10.1186/s12913-015-1017-7
      OpenUrl
    32. ↵
      1. Oremus M,
      2. Oremus C,
      3. Hall GBC, et al
      . ECT, Team CSR. Inter-rater and test–retest reliability of quality assessments by novice student raters using the Jadad and Newcastle–Ottawa Scales. BMJ Open 2012;2:pii: e001368. doi:10.1136/bmjopen-2012-001368
      OpenUrl
    33. ↵
      1. Jiwa M,
      2. Mathers N,
      3. Walters S
      . Quality of information on referrals to colorectal surgeons: towards consensus. Curr Med Res Opin 2002;18:72–7. doi:10.1185/030079902125000309
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Huskisson EC
      . Measurement of pain. Lancet 1974;2:1127–31.
      OpenUrlPubMedWeb of Science
    35. ↵
      1. Larsen S,
      2. Aabakken L,
      3. Lillevold PE, et al
      . Assessing soft data in clinical trials. Pharm Med 1991;5:29–36.
      OpenUrl
    36. ↵
      1. Ree AO
      . KITH Rapport 22/03 Medisinsk- faglig innhold i henvisninger- “Den gode henvisning”. Trondheim: KITH: Kompetansesenter for IT i helsevesenet AS, 2003:35.
    37. ↵
      1. Akbari A,
      2. Mayhew A,
      3. Al-Alawi MA, et al
      . Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev 2008;(4):CD005471. doi:10.1002/14651858.CD005471.pub2

    Footnotes

    • Contributors SLE, TdL and LA designed the study. SLE, KW, ØH, BS, F-AH, TO, ES, KG and GH-H performed the data collection. SLE and CB performed the power and data analysis. SLE drafted the paper. All authors critically reviewed and improved it. SLE is the guarantor. All authors had access to all the data and take responsibility for the integrity of the data and the analysis.

    • Funding The PhD student salary was funded by the South-Eastern Norway Health Authority's research grant (grant agreement number 2008040) and the Norwegian Medical Association grant for quality improvement and patient safety (grant agreement number 14/1689). Researchers were independent of the funder.

    • Competing interests None declared.

    • Ethics approval The study was reported to and approved by the Data Protection Official for research. The Regional Ethics Committee considered the study outside its mandate, and its approval was not required. The presented data are anonymised and risk of identification is low.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The full data set and statistical code can be made available from the corresponding author.