Article Text
PDF
Abstract
Objectives To assess the nature, quality and independence of scientific evidence provided in support of claims in industry-authored educational materials in oral health.
Design A content analysis of educational materials authored by the four major multinational oral health product manufacturers.
Setting Acute care settings.
Participants 68 documents focused on oral health or oral care, targeted at acute care clinicians and identified as ‘educational’ on companies’ international websites.
Main outcome measures Data were extracted in duplicate for three areas of focus: (a) products referenced in the documents, (b) product-related claims and (c) citations substantiating claims. We assessed claim–citation pairs to determine if information in the citation supported the claim. We analysed the inter-relationships among cited authors and companies using social network analysis.
Results Documents ranged from training videos to posters to brochures to continuing education courses. The majority of educational materials explicitly mentioned a product (59/68, 87%), a branded product (35/68, 51%), and made a product-related claim (55/68, 81%). Among claims accompanied by a citation, citations did not support the majority (91/147, 62%) of claims, largely because citations were unrelated. References used to support claims most often represented lower levels of evidence: only 9% were systematic reviews (7/76) and 13% were randomised controlled trials (10/76). We found a network of 20 authors to account for 37% (n=77/206) of all references in claim–citation pairs; 60% (12/20) of the top 20 cited authors received financial support from one of the four sampled manufacturers.
Conclusions Resources to support clinicians’ ongoing education are scarce. However, caution should be exercised when relying on industry-authored materials to support continuing education for oral health. Evidence of sponsorship bias and reliance on key opinion leaders suggests that industry-authored educational materials have promotional intent and should be regulated as such.
- oral medicine
- medical ethics
- health policy
- health services administration & management
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
We sampled all documents explicitly labelled as ‘educational’ from the websites of the four major manufacturers of oral care products.
All data were extracted in duplicate and judgements about whether evidence substantiated a claim was made by two independent reviewers.
We included a novel evaluation of the independence of the cited evidence by assessing relationships among cited authors and the manufacturers.
We do not know whether or how these educational materials are used by clinicians and thus the impact on practice is unknown.
Introduction
Industry continues to be a major source of sponsorship of clinicians’ continuing education in the form of conferences, dinner meetings, journal clubs, grand rounds and trainings.1 2 Nurses frequently rely on industry representatives and information for guidance on product use and outcome evaluation in the practice setting.3–5 Products commonly used in nursing care, such as wound dressings, often lack high-quality clinical trials demonstrating efficacy prior to market approval.6 7 Thus, manufacturers are often a principle—or sole—source of information about nursing-related products.
However, information communicated to health professionals about pharmaceuticals and devices in the form of product advertisements often fails to provide adequate safety information, or to communicate an appropriate balance between benefits and harms.8–10 Less is known about the nature, quality or impact of industry-authored materials that are characterised as ‘educational.’ Educational materials in many jurisdictions are not subject to the same regulation as advertising, thus, may not undergo regulatory review for inclusion of appropriate safety information, for example.11 Thus, the goal of this study was to evaluate the nature and quality of industry-authored educational materials from the perspective of evidence-based practice.
We selected oral health in acute care settings as the case study for this analysis for three reasons. First, oral diseases affect over half of the world’s population, including untreated dental caries, which globally, is the most prevalent health condition.12 Second, oral health represents an opportunity to examine a variety of commercial determinants of health as it is characterised largely by a downstream, interventionist and technology-focused approach.13 Third, inadequate oral hygiene represents a serious risk factor for healthcare-acquired pneumonia, which is an important source of morbidity, mortality and growing healthcare costs.14 Thus, there is increased interest by hospital administrators and health systems in addressing patients’ oral health, which has placed a spotlight on the selection and use of efficacious tools and pharmaceuticals for oral care.
Consequently, nurses face increasing expectations to deliver safe and effective oral care.14 15 Oral care is a fundamental care practice for which nurses are primarily accountable and occurs within complex clinical and technical environments in order to prevent associated adverse health and quality of life outcomes including pneumonia, painful oral diseases such as periodontitis and tooth loss.14 16 17 However, nurses consistently experience insufficient pre-licensure and post-licensure education in oral healthcare,18 which is consistent with the siloing of oral health by health systems, policymakers and medicine more broadly.13 Given these educational gaps, in this content analysis, we focus on educational materials authored by the manufacturers of products used to perform oral care in acute care hospital settings including toothbrushes, foam swabs, lip moisturiser, oral rinses and oral suction (see online supplemental table 1). We aimed to assess the nature, quality, and independence of scientific evidence provided in support of product-related and practice-related claims.
Supplemental material
Materials and methods
Design and sampling frame
We identified manufacturers of oral care products (see online supplemental table 1) through expert consultation (CMD), previous research on nurse-industry interactions,4 Google searches for oral care product brands and examination of the regulatory filing (SEC 10-K form) for the dominant manufacturer (Sage), which identified the major competitors in the company’s medical division. We excluded companies that were at the start-up phase or supported exclusively through grants, and that only distributed and did not manufacture oral health products. Our sampling frame thus included educational materials authored by:
Sage Products (publicly traded manufacturer, a subsidiary of Stryker, a Fortune 500 company, USA, manufacturer of Q·Care Oral Cleansing & Suctioning Systems).
Medline Industries (privately held manufacturer and distributor, USA, manufacturer of Medline brand toothbrushes, swabs, Yankauers, mouthwashes and DenTips Oral Swabsticks).
Intersurgical (privately held manufacturer, UK, manufacturer of OroCare 24-hour day kits).
Avanos (publicly traded manufacturer, USA, manufacturer of Ballard Oral Care kits).
Data sources
Two investigators independently sampled all educational materials from the four companies’ international websites; thus, all content was in English. We defined ‘educational material’ as documents produced and authored by the company, focused on oral health conditions and/or care practices, targeted at clinicians, and explicitly identified as ‘educational’ (eg, located under website headers ‘clinical education,’ or identified as a ‘course’ or ‘training’). There were no restrictions on document format. We captured screenshots of all included web pages and downloaded all available PDFs. Two investigators independently screened the full texts of sampled documents according to these inclusion criteria with a third investigator reviewing any discrepancies. We excluded documents if they were required by a regulator (eg, Material Data Safety Sheet), intended for purchasing (eg, catalogue, order form), hosted and/or authored exclusively by a third party, or targeted patients, family caregivers or clinicians working outside of acute care (eg, dentists).
Data extraction
Based on previous analyses of evidentiary support for promotional claims in pharmaceutical and medical device advertising,8 9 19 we created a data extraction tool in REDcap20 that comprised three main sections: identification of products, identification and assessment of product-related claims, and identification and assessment of supporting evidence. Identification of products included assessing the number and type of unique products mentioned or depicted. We extracted all product-related or practice-related claims, defined as statements made about the efficacy, safety, cost-effectiveness, convenience, or other value of an oral care product (eg, toothbrush) or clinical practice involving a product (eg, toothbrushing), along with any accompanying citation(s). We distinguished product-related claims from normative claims, which suggested what should or must be done, but did not refer to effectiveness, for example.
We categorised claims using an adapted typology from a previous investigation of pharmaceutical advertisements8: unambiguous (ie, clinical comparison or outcome that is clear and measurable); vague or non-clinical (ie, lacks a comparison, clear efficacy outcome or clinical outcome); process-related (ie, related to workflow, convenience or compliance concerns); and emotive/immeasurable (ie, evoked feelings and no measurable outcome identified) and noted whether the claim contained risk reporting.
We extracted all citations, then classified citations accompanying claims by type (eg, journal article, conference abstract, data on file) and level of evidence according to the criteria for treatment efficacy from the Oxford Centre for Evidence-Based Medicine.21 We determined whether a citation identified a primary outcome and data were extracted on the citation’s funding sources and author conflicts of interest.
We piloted the instrument on a subset of sampled documents until we reached an acceptable level of agreement. Two investigators then independently extracted data on the entire sample; discrepancies were discussed and resolved with a third author.
Data analysis
Two independent investigators assessed claim–citation pairs, which involved a claim and accompanying citation, to determine if information in the citation supported the claim). Investigators classified citations deemed ‘unsupportive’ according to an adapted classification from a study of claim–citation pairs in wound care advertising,9 choosing the reason that best described why the citation was unsupportive. Reasons included: the citation was unrelated in terms of content, study population or intervention; exaggeration of benefits; citation reported an in-vitro or animal study; distorted reporting of study findings (eg, the claim was not based on the study’s primary outcome, the study findings were not statistically significant or the citation did not meet an appropriate level of evidence for the accompanying claim) or cited data were unpublished (eg, ‘data on file’). We calculated descriptive statistics on all frequencies and proportions using SPSS V.25.
Network analysis
In addition to the level and quality of evidence used to substantiate claims, we assessed the independence of the evidence presented using social network analysis. We sought to analyse two facets of independence: (1) the degree to which industry-authored educational materials cited the work of authors who work independently from one another (ie, authors who are not coauthors); and (2) the extent of referenced authors’ relationships with the sampled companies and industry more broadly.
We manually extracted the listed authors and coauthors for all publications referenced in the sample, excluding sampled documents with no citations and non-authored citations (eg, data on file, federal register, no listed authors). We calculated the number of times each publication was cited in substantiation of a claim and the number of times each publication was cited overall. Then, we ranked authors by the number of cited publications they authored or coauthored in substantiation of a claim. To analyse the interdependence of authors, we derived the network of coauthorship relations derived from these references.
Patient and public involvement
There was no patient or public involvement in this study.
Results
We included 68 documents from the four manufacturers (figure 1). Nearly 2/3 (43/68, 64%) were authored by Sage (owned and operated by Stryker Corporation), the dominant manufacturer in this market. Document characteristics are outlined in table 1. Sampled documents included brochures, flyers, web pages and courses containing information about oral care (eg, ‘Evidence-based practices for comprehensive oral care workshop’), oral disease (eg, ‘Colonisation of dental plaque and importance of brushing for hospitalised patients’), or sequelae of missed oral care or oral disease (eg, ‘Protecting your patients from ventilator-associated pneumonia’). Sampled documents also included templates for educational posters, and oral care assessment or care protocols designed to be customised by users. The majority of documents mentioned an oral care product (59/68, 87%) and 51% mentioned a branded oral care product (35/68), which included pharmaceuticals (eg, oral rinse), medical devices (eg, toothbrushes, suction devices) or pre-packaged kits containing a combination of oral care products and pharmaceuticals (online supplemental table 1). The majority of documents contained at least one product-related claim (55/68, 81%). We extracted 252 claims across the sampled documents; however, claims frequently recurred verbatim across the 68 documents, resulting in 204 unique claims (204/252, 79%).
Industry-authored educational materials sampling flow diagram (n=68).
Characteristics of industry-authored educational materials (n=68)
Evidentiary support for claims
The majority of claims (124/204, 61%) referred to an outcome that was vague and/or non-clinical (see table 2). Only 12% (24/204) of claims contained risk reporting; on examination of the accompanying citation, we determined the majority of claims containing risk reporting (18/24, 75%) reported relative risk, while 6 (25%) did not present sufficient information to determine the type of risk reporting.
Nature of outcome reporting in claims
Of the 204 unique claims, 56% (115/204) were accompanied by one or more citations, resulting in 147 unique claim–citation pairs. For the majority of claim–citation pairs, we judged the claim to be unsupported by the accompanying citation (91/147, 62%). Most often, citations did not provide adequate support for the claim because citations were unrelated in terms of content focus, study population or intervention; the underlying evidence was inaccessible to a frontline clinician; or claims exaggerated the benefits of the cited findings. Table 3 provides illustrative examples of citations that provided insufficient support to claims.
Nature of evidentiary support or non-support of claims
Nature and level of evidence
Documents referenced a mean 6.62 citations (SD=11.89). We extracted 437 citations from the 68 documents; 31% of the citations (134/437) appeared in multiple documents, resulting in 303 unique citations in the sample of 68 documents (table 4). However, the majority of unique citations (71%, 215/303) accompanied statements unrelated to oral health or general statements of fact (eg, ‘Every 4–6 hours 20 billion bacteria duplicate in the oral cavity’). Only 29% (88/303) of unique citations occurred as part of a claim–citation pair. We were unable to identify or access the full text of 14% (12/88) because citations were incomplete (eg, American Association of Critical-Care Nurses Manual, 2015) or data were unpublished (eg, data on file with manufacturer, presentation abstracts and proprietary reports). Thus, we categorised 76 citations by level of evidence. Cited studies generally represented lower levels of evidence: less than 20% were systematic reviews (7/76, 9%) or randomised controlled trials (10/76, 13%). About half the cited studies provided a conflict of interest statement (43/76, 57%) and/or a funding statement (36/76, 47%). Of the cited studies that made such disclosures, 23% (10/43) disclosed financial relationships between authors and oral health product manufacturers, 33% (12/36) reported industry sponsorship of the study; two studies reported both author conflicts of interest and industry funding for the study.
Characteristics of cited studies
Independence of evidence
We identified 796 unique authors of citations referenced in the sampled documents; 38% (304/795) were authors of citations used to substantiate a claim. Using social network analysis, we examined the degree to which authors of citations accompanying claims were independent from one another (ie, authors who are not coauthors). Within sampled documents, a small group of individuals authored and coauthored a disproportionate number of citations used to substantiate claims.
Figure 2 displays the coauthor network derived from citations used to substantiate a claim within sampled documents. The nodes represent individual authors, joined by ties that indicate they coauthored at least one citation in the sample. The size of the node represents the number of citations the individual authored within the sample that were used to substantiate claims. Nodes coloured dark blue highlight the top 20 authors ranked by the number of citations; light blue nodes indicate authors that are directly or indirectly linked (through shared coauthors) to the top 20 authors.
Network of authors and coauthors referenced by claims: the nodes represent individual authors, joined by ties that indicate coauthorship. The size of the node represents the number of citations the individual authored within the sample that were used to substantiate claims. Nodes coloured dark blue highlight the top 20 authors ranked by the number of citations; light blue nodes indicate authors that are directly or indirectly linked (through shared coauthors) to the top 20 authors.
These top 20 authors occupied central positions in the network, connecting and collaborating with many of the author groups whose work companies cited to provide an evidence base for the educational materials. The top 20 authors (in terms of the number of times their authored or coauthored citations were used to substantiate a claim) represented 2.5% of all authors in the overall sample of cited authors (20/796). Collectively, they accounted for 37.4% of all citations used within claim–citation pairs (n=77/206, including claim–citation pairs repeated across documents) (table 5).
Characteristics of top 20 authors
We investigated the industry ties of these top 20 authors (table 5). Overall 60% (12/20), including the top five authors, had at least one financial relationship with one of the four sampled oral health product manufacturers, which included receipt of personal payments for speaking or consulting and/or study funding. Among these top 20, only 1 author (5%) had no financial ties to industry.
Discussion
Oral health product manufacturers authored a wide range of educational materials targeted at nurses ranging from product training videos to courses. However, these educational materials may be largely characterised as ‘education in support of a product’4: the majority mentioned an oral health product, half mentioned a branded product and over 80% made a product-related claim. Given that oral health is the product of a complex interplay among social (eg, socioeconomic status, marginalisation, access to dental care) and commercial determinants (eg, promotion of high-sugar products),12 the educational focus on product-related practices suggests a downstream approach to oral health and may constitute an agenda bias in educational content and the underlying research.22
Educational materials authored by these companies presented as evidence-based, containing on average nearly seven citations per document and suggested they represented the findings of curated scientific literature (ie, titles such as ‘What the Experts Say’). Just over half of the unique claims (115/204, 56%) were accompanied by a citation and the majority were not substantiated by the underlying evidence. In general, sampled documents presented a low level of evidence and relied heavily on narrative reviews or opinion pieces; however, most claims related to vague or non-clinical outcomes, thus, the level of evidence required to support such statements is also lower. Commonly, claims presented a distorted interpretation or exaggerated the benefits of the accompanying evidence, which constitutes a form of ‘spin,’ defined as reporting practices that mislead readers by presenting results in a more favourable light.23
The companies relied on a small network of oral health experts in marshalling evidence in support of claims and educational materials more generally, many of whom had existing or subsequent financial ties to the companies or industry more broadly. These recognised and respected experts are examples of key opinion leaders, who are engaged by pharmaceutical or medical device companies as speakers or consultants for their ability to influence their peers.24 Companies may also approach key opinion leaders to serve as investigators on company-sponsored projects or as authors on company-led research.25 Key opinion leaders are valuable to companies because they project an appearance of independence and integrity, while serving as ‘product champions’; however, companies carefully manage key opinion leaders, including nurses, physicians and scientists, through training programmes and by offering targeted research funding, speaking platforms and authorship opportunities.24
Companies also sponsored or were involved in nearly half of the highly cited studies suggesting sponsorship bias, where industry funding is associated with results and conclusions favourable to the sponsor,26 may also be of concern. Regardless of the educational value and integrity of the underlying research, our network analysis illustrates how companies can strategically cite, often repeatedly, and thus amplify, perspectives that are favourable to commercial aims. This may be another facet of sponsorship bias consistent with previous research that found articles with positive conflict of interest disclosures are more likely to be published in high-impact journals or to receive more media attention.27
Consequently, industry-authored educational should be characterised as ‘promotional’ and regulated as advertising. Regulators have issued industry guidance to enable assessment of the distinction between ‘promotional’ and ‘non-promotional’ activities, which includes assessing whether materials directly or indirectly promote the sale of a health product and whether the manufacturer or sponsor has influence over the content.11 28 In practice, however, medical device industry-authored educational materials likely receive little regulatory scrutiny. Though certain high-income countries such as Canada, Australia, the USA and the European Union have specific laws that govern pharmaceutical and medical device advertising, these regulators are under-resourced and most jurisdictions rely on voluntary, industry self-regulation through codes of practice to regulate promotion.29 30
Strengths and limitations
We analysed a purposive sample of publicly available educational materials sampled from the websites of four manufacturers of oral health products. It is unknown whether these documents are representative of those produced by other oral health manufacturers, nor whether these findings can be generalised to other product categories. However, the sampled companies are market leaders and two (Sage and Medline) have diverse product portfolios suggesting that these findings may be indicative of industry-authored educational materials more broadly. We sampled educational documents targeting nurses from company websites, thus it is unknown whether and how these educational materials are used and their impact on educational or clinical outcomes. Identifying educational materials and extracting claims required interpretation, thus we opted for duplicate sampling and data extraction at all stages.
Conclusion
The sustainability of health systems worldwide is under strain and resources to support nurses’ ongoing practice-based education are scarce. The findings of this study, however, suggest that caution should be exercised when relying on industry-authored educational materials to support product training and continuing clinical education in oral health and in clinical practice, more broadly. To support the use of oral health products in clinical practice, clinicians should seek industry-authored materials that conform to regulatory standards related to labelling (ie, instructions for use) and otherwise, seek education that is independent from manufacturers.
The findings of this study call into question whether industry-authored materials are educational or promotional, which carries regulatory implications. Evidence of sponsorship bias affecting the focus, substantiation of claims and curation of expert recommendations suggests that industry-authored educational materials have promotional intent and should be regulated as such.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @QuinnGrundy, @craig_dale1
Contributors QG conceived the study, acquired funding, supervised all aspects of the work and wrote the first draft; AM and CC conducted sampling, data collection and analysis, and critically revised the manuscript; FH conducted network analyses, performed data visualisation and critically revised the manuscript; CMD contributed to study design, supervised all aspects of the work and critically revised the manuscript.
Funding This work was supported by the Bloomberg Summer Research Scholarship (no award number) from the Lawrence S Bloomberg Faculty of Nursing at the University of Toronto, the Connaught New Researcher Award (no award number) from the University of Toronto and the Toronto Mobility Scheme (no award number) from the University of Sydney’s Office of Global Engagement.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement The full dataset is available as a CSV file from the authors upon reasonable request.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.