Study design

To achieve the objectives of this study, it was decided to search the Web of Science database of Clarivate Analytics for highly cited articles and track the citation records of publications identified. Although Scopus and Google Scholar databases also provide citation tracking, it was decided to limit the search to the Web of Science. This is because the Web of Science is regularly updated and its 2016-Journal Citation Reports reported over 59 million citations in its Science Edition and 7 million from its Social Science Edition. In the area of medical education, medical ethics and bioethics, general medicine and surgery, the Web of Science has included 16, 49, 457 and 180 peer-reviewed journals, respectively. Google Scholar was not included in the search because it is difficult to search, and it is not possible to identify the number of citations for each year across the last two to three decades, and the citations in Google Scholar usually include textbooks, monographs, conference proceedings, as well as non-peer-reviewed work. The Scopus database was not included in our search because its records only go back to 1966.

To achieve the first aim, we planned to identify the highly cited articles in medical professionalism and their characteristics using three mechanisms: (1) searching the Web of Science using keywords, (2) searching medical education, ethics, general medicine and surgery journals in the Web of Science, (3) searching the webpage of journals and (4) searching for related resources mentioned in the list of references of articles identified. For the second aim, the altmetric bookmarklet application was used to obtain the altmetric scores and construct exploratory analysis examining the role of social media and the different resources contributing to altmetrics. At the end, we compared these findings with those obtained from the citation analysis.15–17 A description of the steps used in the search is discussed below.

Searching the web of science database using keywords

Searching the Web of Science database was carried out in the 5 April 2017 by two researchers (SAA is a professor of medical education with a 20-year experience in research in the field of medical education and professionalism, and SA a surgical registrar and researcher). The search words used were the following: ‘Medical professionalism’, ‘Patient safety’, ‘Professional behavior’, ‘Unprofessional behavior’, ‘Role modeling’, ‘Accountability’, ‘Faculty training in professionalism’, ‘Altruism’, ‘Physician code’, ‘Physician charter’, ‘Medical ethics’, ‘Integrity’, ‘Consent’, ‘Defining medical professionalism’, ‘Empathy’, ‘Compassionate doctor’, ‘Professional conduct’, ‘Collaborative doctor’, ‘Self-assessment’, ‘Professional development’, ‘Resilient doctor’, ‘Social justice’, ‘Patient autonomy’, ‘Patient Welfare’, ‘Professional responsibility’, ‘Managing conflict’, Patient confidentiality’, ‘Quality of care’, ‘Social contract’, ‘Team work and professionalism’, ‘Personal development’, ‘Public professionalism’, ‘Interpersonal professionalism’ and ‘Intrapersonal professionalism’. These keywords were identified from the terminology and themes used in defining medical professionalism in six resources including.18–23 We also looked at conference proceedings in the field and websites of organisations and agencies responsible for accreditation of medical education worldwide including: The World Federation for Medical Education, the UK’s General Medical Council, the Association of American Medical Colleges, the Australian Medical Council, the Liaison Committee on Medical Education, and the Quality Assurance of Basic Medical Education, and documents such as: Tomorrow’s doctors, 2003; The New Doctor, 2004; and General Medical Practice, 2001.

For each search word, the results were arranged using a link on the Web of Science database system ‘sort-by’—‘Time Cited- highest to lowest’. The results showed the articles organised in a descending order with the articles most frequently cited on the top. The findings from each search word were then arranged on one Excel sheet in a descending order based on the number of citations. The results identified by each evaluator were discussed and duplicate articles were excluded.

Searching journals in the Web of Science

The second search involved searching all journals in the field of medical education, ethics, general medicine and surgery included in the Web of Science database. These journals are known to publish articles on medical professionalism. They were selected on the basis of the outcomes of the Web of Science search and the references cited by the articles identified. The aims of this second search were to maximise the yield of the search and detect any articles that were possibly missed during the first search. This search was conducted under the same conditions of the first search, by the two researchers, on the same day, and by using the same keywords used in the first search. The journals in medical education that were searched included Academic Medicine, Medical Education, Medical Teacher, BMC Medical Education, Advances in Health Sciences Education Theory, and Practices, Teaching and Learning in Medicine and the Journal of Continuing Education in the Health Professions. The journals searched in general medicine and surgery were the New England Journal of Medicine, the Lancet, the British Medical Journal, the Journal of the American Medical Association, Journal of General Internal Medicine, Annals of Internal Medicine, Archives of Internal Medicine, Canadian Medical Association Journal, PLOS Medicine, Annals of Surgery, Archives of Surgery, British Journal of Surgery, Perspectives in Biology and Medicine, Mayo Clinic Proceedings and the Australian Medical Journal. The journals in bioethics that were searched included the American Journal of Bioethics, Journal of Medical Ethics and BMC Medical Ethics. The findings from journals were then arranged on one Excel sheet in a descending order based on the number of citations. The results identified by each evaluator were discussed and duplicate articles were excluded.

Searching the webpage of journals

To maximise the yield of our search and to ensure that no paper was missed through searching the Web of Science, we conducted a third search using the webpage of the journals mentioned above. We examined the titles of articles listed in each issue of these journals during 2011 and prior years. This search was particularly important as for example, Teaching and Learning in Medicine first appeared in Web of Science in 1996 but the journal was published since 1989. Therefore, any relevant articles from this journal or others prior to 1996 would be included.

Inclusion and exclusion criteria

The inclusion criteria were: (1) papers focusing on medical professionalism in the English language and (2) articles, reviews, research papers, reports, editorials on any aspect related to medical professionalism in the English language. The exclusion criteria were: (1) articles on medical professionalism in languages other than English, and (2) articles that focused on education/curriculum or clinical practices and medical professionalism was not the main focus. Articles with identical absolute number of citations were ranked on the basis of the average citation per year (the number of citations obtained divided by the number of years since published).24 A copy of all papers included in the list was obtained and read by the evaluators.

It is interesting to note that none of the articles excluded on the basis of language were qualified for inclusion in the list because they had less citation numbers than those of the article marked number 50 in the list.

Assessing the articles

For each of the identified articles (online supplementary appendix 1), a full text was obtained and a copy was given to each researcher. The following information was collected: (1) the authors’ names and their affiliations, and the number of females contributing to authorship (2) the number of institutes involved and the city and country of the origin of the publication, (3) the total number of citations obtained up to the day of searching the database, and the number of yearly citations since publication, (4) the year of publication and the calculated number of years since publication and (5) grants/funding bodies stated in the publication and (6) the 2016-JIF of the journal that published the work.

We also aimed at grouping the identified top-cited articles into categories. We have not used the categories provided by the Web of Science ‘study type’ because we noted that the Web of Science system does not differentiate between ‘original research’ or ‘articles’ and classified both as ‘articles’. For consistency and the purpose of this study, we grouped the articles into four categories—article, review, editorial material and research. A definition of each category is given in the glossary. Using these definitions, two researchers independently allocated each article under a category. For articles that were difficult to classify or not fitting into the same category, a meeting was held to discuss these articles and a final decision was made.

The topics covered in identified articles were created by each researcher independently by generating key words reflecting the main idea covered in an article and using these words to phrase a short statement that could help in grouping more than one article under one topic. The topics were then discussed in a meeting to harmonise the grouping into a logical, simple and practical approach. Articles covering more than one topic were classified on the basis of the aim of the study, the title and the main outcomes.

Identification of author’s gender

Regarding the data collected for each article, it is important to mention here that the identification of the gender of each author was a challenging task particularly when a journal uses abbreviations of the first and second name rather than the full name, which was the case in three articles. The approach used in order to identify the females in the top-cited articles included (1) searching the Google database to find the university website, personal website of the author, LinkedIn webpage and ResearchGate account. This approach was particularly useful for authors who could have moved to other universities, (2) searching the university websites not only provided the full names but also provided identification photos of these authors, and in many times a list of their publication records, as well as areas of research/teaching interests (3) Searching the Google Scholar database to identify their accounts, where we can find other publications under their names, the full name or an identification photo showing them. Usually, authors of highly cited articles have other publications related to the same topic, or work with the same coauthors, which could also help in identifying them and tracing them and (4) In two difficult cases we emailed the corresponding author of these articles for help.

Altmetric system

The altmetric system comprises, but not limited to, policy documents, news, blogs, tweeters, online reference managers (eg, Mendeley, CitULike), postpublication peer reviews (eg, Publons), Social media platforms (eg, Facebook, Google+, Pinterest), citations on Wikipedia, sites running Stack Exchanges (Q&A) and reviews on Faculty 1000 (F1000) and YouTube. Therefore, altmetric scores may reflect interest of the public as well as clinicians and researchers in a publication and the scores may provide information about the geographical and demographic details of those involved in such online/social media discussions.25

The altmetric programme processes raw data collected from the above-mentioned resources and the data are weighted according to a system created by altmetrics to reflect the relative contribution of each source to the total altmetric score. News, Blogs, Wikipedia and policy documents have a relatively higher weighting values.26 While Mendeley and CiteULike are shown in the report, they do not contribute to the total score.

Searching the altmetric system

The search of the altmetric system was conducted on the same day (5 April 2017). The scores were identified using the Altmetric bookmarklet provided by the company.27 In summary, the articles were searched on PubMed database (the PMID or DOI are essential for triggering the altmetric bookmarklet to function). By clicking on the LinkOut link, we identified the publisher webpage hosting the original article and by clicking the altmetric bookmarklet application, we can check the attention records for the article. The altmetric attention score and donut help in identifying the relative quantity and the type of attention received by a published article. The meaning of the colours included in altmetric donut is explained in this link.28

Statistical analysis

All analyses were conducted using SPSS Software (IBM SPSS Statistics Premium V.22.0 for Mac OS-SPSS) and the results were reported at total, mean, median, IQR and percentage. Pearson’s correlation coefficient (r) was calculated to determine if the high citation numbers obtained were related to parameters characteristic of articles. Because of the observed differences in the citations of the top articles in the list compared with those in the bottom of the list, and the variability in the altmetric scores, it was decided to conduct a multivariate analysis model comprising the effect of number of authors and other parameters. The inter-rater agreement between evaluators was calculated using the Fleiss kappa scale.29

Patient and public involvement

Patients and the public were not involved in this study.

Top-cited papers identified

Online supplementary appendix 1 summarises the 50 most-cited articles in medical professionalism identified by searching the Web of Science database,30–79 out of a total of 3500 articles identified on professionalism. The articles are listed in a descending order from 1 to 50 with the highest absolute citation number is ranked 1 and the article with the lowest citation ranked 50 as per the day of the search. Articles with the same number of citations were ranked on the basis of average citation per year (eg, the articles ranked 34 and 35 had the same citation number 97, they were allocated to a ranking order based on the calculated citation per year, 13.86 and 7.46, respectively). Other articles that had the same citation number and were ranked on the basis of their calculated citation per year were articles ranked 36 and 37; 43 and 44; as well as 46 and 47.

Table 1 summarises the year of publication and article category. The articles were published over 17 years (from 1994 to 2011). During the period from 1994 to 1999, only seven articles (14%) were published. However, the number increased significantly from 2000 to 2005 making a total of 24 (48%) articles. The number in the years from 2006 to 2011 dropped to 19 (38%). No correlation was found between the citation counts of these papers and the number of years since publication (Pearson correlation (r)=0.188, p=0.192).

The distribution of the medical professionalism topics covered in these articles is summarised in table 2. The inter-rater agreement between assessors was in the range 0.758–0.846.

The articles were published in the following journals: Academic Medicine (n=19, 38%), the Journal of the American Medical Association (n=9, 18%), Journal of General Internal Medicine (n=4, 8%), Annals of Internal Medicine (n=4, 8%), the New England Journal of Medicine (n=3, 6%) and Medical Education (n=3, 6%). It is interesting to note that 24 (48%) articles were published in eight general medicine journal and the remaining were published in four medical education journals and one journal specialised in bioethics. Most journals have high journal impact factors and are on the top of their field (table 3). This finding reflects the significance of medical professionalism in undergraduate and postgraduate training. The first author of the top-cited articles was from the USA (n=37, 74%), Canada (n=8, 16%), the UK (n=2, 4%), Germany (n=1, 2%), Israel (n=1, 2%) and New Zealand (n=1, 2%).

Table 4 summarises the 26 authors who have contributed to two or more articles in the list. Of these, five authors were the first authors of two or more papers, 10 were coauthors of two or more papers, and the remaining 11 were the first authors and coauthors of two or more articles. Top authors were Papadakis, M (n=4; first author of all four papers) and Blank, L (n=4; first author of one paper and coauthor of three). Other top authors are shown in table 4.

The leadership of universities and institutes that have contributed to the creation of these publications are shown in online supplementary appendix 1.

Characteristics of the top-cited articles

These articles were created by 252 authors, median 4, minimum 1, maximum 19, IQR 4 and the females in authorship were 102, median 2, minimum 0, maximum 11, IQR 2. The institutes involved were 168, median 2, minimum 1, maximum 17, IQR 3; the countries involved were 67, median 1, minimum 1, maximum 9, IQR 0; and the grants/funds received were 35, median 0, minimum 0, maximum 8, IQR 1. Significant correlations were found between the number of citations and the 2016-JIF (Pearson correlation (r)=0.318; p=0.024), and the number of countries (r=0.453; p=0.001). No significant correlations were found between the number of citations and the number of years since publication (r=0.188, p=0.192), the number of authors (r=0.159; p=0.270), the number of females in authorship (r=0.343; p=0.150), the number of institutes involved (r=249; P=0.081) or the number of grants received (r=−0.126; p=0.384).

The altmetric scores

The altmetric scores and reports were found for 70% of articles. The total scores were 806, median 5, minimum 0.0, maximum 155, IQR 20. No correlation was found between the number of citations and the total altmetric scores (r=0.064; p=0.661). A significant correlation was found between number of citations and altmetric scores for articles published in 2007 and after (n=17, r=0.547, p=0.023). No correlation was found for articles published in 2006 or earlier. Only 38% of the articles had readers on CiteULike (mean 1.6, 95% CI 0.4 to 2.7, median 0, minimum 0, maximum 19, IQR 1; while 62% were read Mendeley (mean 72.7, 95% CI 45.2 to 100.3, median 39.5, minimum 0, maximum 499, IQR 120. The coverage of journal articles by Twitter was 54% (mean 7.8, 95% CI 2.7 to 13.0, median 1.5, minimum 0, maximum 117, IQR 10) followed by blogs 38% (mean 1.2, 95% CI 0.6 to 1.8, median 0, minimum 0, maximum 7, IQR 1, then policy sources 24% (mean 0.38, 95% CI 0.15 to 0.61, median 0.0, minimum 0.0, maximum 3, IQR 0, then Facebook 20% (mean 0.3, 95% CI 0.07 to 0.61, median 0, minimum 0, maximum 6, IQR 0). The Wikipedia was the lowest resource. No significant correlation was found between the number of citations and altmetric scores (r=0.064; p=0.661). The geographical breakdown showed that USA had the highest share, followed by UK. Other countries identified for some articles were Canada, Mexico, Spain, Australia, Spain, Chile, Netherlands, Portugal, Japan, Columbia, Italy, France and Brazil. We looked at journals’ webpages of the top-cited articles and those hosting the altmetric metre; we did not find significant correlation between hosting the altmetric metre and recorded altmetric scores.

Multivariate analysis

Because of the observed differences in the citations of the top articles in the list compared with those in the bottom of the list, and the variability in the altmetric scores, it was decided to conduct a multivariate analysis comprising the effect of number of authors, number of institutes, number of countries, number of females in authorship, number of grants obtained on the citation scores and altmetric scores (table 5).