Article Text


Comparing human papillomavirus vaccine concerns on Twitter: a cross-sectional study of users in Australia, Canada and the UK
  1. Gilla K Shapiro1,2,
  2. Didi Surian3,
  3. Adam G Dunn3,
  4. Ryan Perry2,
  5. Margaret Kelaher4
  1. 1 Department of Psychology, McGill University, Montreal, Canada
  2. 2 Centre for Health Policy, School of Population and Global Health, University of Melbourne, Melbourne, Victoria, Australia
  3. 3 Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, New South Wales, Australia
  4. 4 Head Evaluation and Implementation Science, Centre for Health Policy, School of Population and Global Health, University of Melbourne, Melbourne, Victoria, Australia
  1. Correspondence to Gilla K Shapiro; gilla.shapiro{at}


Objective Opposition to human papillomavirus (HPV) vaccination is common on social media and has the potential to impact vaccine coverage. This study aims to conduct an international comparison of the proportions of tweets about HPV vaccines that express concerns, the types of concerns expressed and the social connections among users posting about HPV vaccines in Australia, Canada and the UK.

Design Using a cross-sectional design, an international comparison of English language tweets about HPV vaccines and social connections among Twitter users posting about HPV vaccines between January 2014 and April 2016 was conducted. The Health Belief Model, one of the most widely used theories in health psychology, was used as the basis for coding the types of HPV vaccine concerns expressed on Twitter.

Setting The content of tweets and the social connections between users who posted tweets about HPV vaccines from Australia, Canada and the UK.

Population 16 789 Twitter users who posted 43 852 tweets about HPV vaccines.

Main outcome measures The proportions of tweets expressing concern, the type of concern expressed and the proportions of local and international social connections between users.

Results Tweets expressing concerns about HPV vaccines made up 14.9% of tweets in Canada, 19.4% in Australia and 22.6% in the UK. The types of concerns expressed were similar across the three countries, with concerns related to ‘perceived barriers’ being the most common. Users expressing concerns about HPV vaccines in each of the three countries had a relatively high proportion of international followers also expressing concerns.

Conclusions The proportions and types of HPV vaccine concerns expressed on Twitter were similar across the three countries. Twitter users who mostly expressed concerns about HPV vaccines were better connected to international users who shared their concerns compared with users who did not express concerns about HPV vaccines.

  • Human papillomavirus
  • Vaccination
  • Twitter
  • Health Belief Model
  • Social media.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Strengths and limitations of this study

  • This study conducted an international comparison investigating how human papillomavirus (HPV) vaccination concerns are expressed on Twitter.

  • Machine learning methods were used to identify and classify the proportion and types of concerns expressed in thousands of tweets.

  • The analysis of social connections among Twitter users posting about HPV vaccines in three English-speaking countries (Australia, Canada and the UK) revealed the potential for concerns to spread internationally.

  • While Twitter is used by a substantial number of people, this study is not designed to allow for generalisation to the general population.

  • This study used follower networks to examine social connections, but further research could use other social interactions to measure the spread and impact of negative attitudes about HPV vaccines.


Human papillomavirus (HPV) is a prevalent sexually transmitted infection that can cause cancers and anogenital warts.1–5 Since 2006, three prophylactic vaccines have been developed to protect adolescents from HPV-associated health problems.6 Research has demonstrated that these vaccines are safe and effective in reducing HPV related infections, genital warts and pre-cancers.7–12 As a result, at least 65 countries have implemented HPV vaccination programmes for women in their national immunisation schedules.11

There is notable variation between countries’ HPV vaccine programmes and coverage rates. Australia’s school-based vaccination programme targets girls (since 2007) and boys (since 2013) aged 12–13 years.13 According to Australia’s National HPV vaccination programme register, 85.6% of women and 77% of men received the HPV vaccine (2015 data).14 15 In Canada, all provinces and territories introduced school-based vaccination programmes for girls aged 9–13 years (2007–2010), and six provinces also include boys in HPV vaccine programmes (since 2013).16 According to national parental surveys, 72.3% of women (2013 data) and less than 3% of men received the HPV vaccine (2014 data).17–19 Lastly, the UK only provides a school-based vaccination programme for girls aged 12–13 years (since 2008). According to Public Health England, 89.5% of women in the UK received the HPV vaccine (2015 data).20 HPV vaccine coverage rates are lower than other child or adolescent vaccines in these countries’ national immunisation programmes21–23; suboptimal coverage hinders cancer prevention efforts.24

The media has the potential to dramatically impact vaccine coverage through influencing parental awareness, perception and attitudes.25–29 Unconfirmed reports of adverse events associated with the HPV vaccine published in the media dramatically affected female HPV vaccine coverage in Japan and Colombia.11 30 31 Many individuals use the internet and social media to access health information; however, these sources have been described as a risky platform that can rapidly amplify unbalanced, distorted or inaccurate information about vaccines.25 32–34 For example, a study by Betsch et al found that even 5–10 min of access to vaccine-critical websites negatively influenced individuals’ risk perception and intentions to be vaccinated.35 Similarly, Nan and Madden report that, compared with a control group, participants who were exposed to negative online blogs about HPV perceived the vaccine as less safe, held more negative attitudes and reported a reduced intention to receive the vaccine.36

Previous research has evaluated the public discourse concerning HPV vaccination in newspapers,37–40 online news,41 comments to online news articles,42 Facebook,43 blogs or online forums36 44 45 and YouTube videos.46 47 Twitter is a microblogging service, established in 2006, that has over 313 million users active monthly. Twitter is an important source of information regarding HPV and vaccine hesitancy,32 48 49 and several studies have examined the representation of HPV vaccines on Twitter.33 43 50–54 Though many of these studies analyse a limited number of HPV-related tweets, a few have used data mining and machine learning techniques to analyse a large number of tweets.51 52 54 55 However, no research has conducted an international comparison to evaluate and compare how vaccination concerns are expressed across countries. Furthermore, no research has examined the domestic and international network connectedness of HPV vaccine concern expression.

The aim of this study was to explore the proportion of HPV vaccine concern on Twitter, examine the type of concern expressed in Australia, Canada and the UK and investigate differences in the ways Twitter users connected locally and internationally.


Study overview

Tweets related to HPV vaccines were collected from January 2014 to April 2016 in Australia, Canada and the UK. These countries were selected because they are English-speaking countries, share a similar history and commonwealth membership and their similarity in administering the HPV vaccination in schools. Data captured included information about users’ locations, the text of the tweets and information about social connections. To enable the classification of a large number of tweets, two stages of machine learning classifiers were constructed from a sample of tweets that were manually coded by two investigators.

Study data

Using a similar approach to previous studies that examined large number of tweets in communities of Twitter users posting about HPV vaccines,54 56 the Twitter Search Application Programming Interface (API) was used to collect tweets in the English language about HPV vaccines from January 2014 to April 2016. The search terms were ‘Gardasil’, ‘Cervarix’, ‘hpv AND vaccin*’ and ‘cervical AND vaccin*’. Information extracted from each tweet included the unique tweet identifier, tweet text, creation time, the identifier of the user posting the tweet and geographical coordinates (if available). Without any restrictions applied to the locations of users, the entire dataset included 358 194 tweets (including retweets) by 129 286 users.

A gazetteer was used to transform the text provided by users into coordinates, and any users with self-reported locations that were located in coordinates in Australia, Canada or the UK were included in this analysis (online supplementary material, Section 1).

Supplementary data

Other data that were used in the analyses included the set of social connections formed among the users who were included in the analyses. For each user, the Twitter Search API was used to collect the set of all follower relationships in which the user was involved, shortly after the first time the user posted a relevant tweet in the period. A network was then formed to include all users who tweeted about HPV vaccines from the three countries, and the follower relationships defined the social connections in an unweighted, directed network (online supplementary material, Section 2).

The Macquarie University Human Research Ethics Committee (number 5201401028) and the University of Melbourne’s Research Ethics Board (number 1647488.1) provided ethics approval for data collection and analysis.


Supervised machine learning methods were used to classify the tweets into two stages: (1) to identify tweets that expressed any concern and (2) to classify specific types of concerns. In the first stage, 1000 tweets were sampled from the set of all tweets to manually label those that expressed concerns. In the second stage, 1000 tweets were sampled from the set of tweets that were estimated to be concerns to manually label them by type. The manually labelled tweets were used to train classifiers to label any tweet by the type of concern expressed (online supplementary material, Section 3).

The categories for types of tweets expressing concerns were determined using an inductive and deductive procedure. The Health Belief Model (HBM), one of the most widely used theories in health psychology, was used as the basis for coding the types of HPV vaccine concerns expressed on Twitter.57 The HBM has been used previously to evaluate the determinants of HPV vaccination and non-compliance by identifying perceived susceptibility to HPV, perceived severity of HPV, perceived benefits of HPV vaccination, perceived barriers of HPV vaccination (including tangible barriers such as logistical challenges and psychological barriers such as perceived harms of receiving the HPV vaccine) and cues to action (eg, influences prompting HPV vaccine uptake such as information from healthcare providers, family or friends).47 57–59 To account for additional prominent concerns that were not captured by the model, the constructs were also informed by previous content analyses of media and social media related to the HPV vaccine, as well as literature on vaccine hesitancy.28 33 37 41 42 45 47 60–68 The coding scheme was therefore extended to include mistrust, undermining of religious principles, undermining of civil liberties, additional concerns (not otherwise specified) and ambiguous tweets (table 1). The coding scheme was used by two investigators (GKS and RP) to code 12 types of concerns expressed in a second sample of 1000 tweets. After examining the proportions of different types of concerns in the sample and accuracy of the multiclass classifier (online supplementary material, Sections 3 and 4), types of concerns were combined to improve the performance that could be achieved by the machine learning classifiers (table 1). Combining categories was done based on conceptual similarity and trying to remain as true to the HBM as possible while attaining accuracy of the classifier.

Table 1

Coding scheme for the types of concerns expressed on Twitter

The network of social connections formed by users who posted tweets about HPV vaccines was used to compare the proportions of local (within a country) and international (across countries) followers. The group of users for whom at least half of their relevant tweets were expressing concerns was assigned to one group (concern), and all other users were assigned to another (non-concern). The users were then also split by country, and the proportions of local and international followers were compared across groups.


There were 129 286 Twitter users who posted at least one tweet about HPV vaccines during the period. The location inference method identified 2792 (2.2%) of those users located in Australia, 7237 (5.6%) located in Canada and 6760 (5.2%) located in the UK (table 2).

Table 2

The total number of users and tweets from Australia, Canada and the UK

From the 16 789 users in the three countries, a total of 43 852 tweets about HPV vaccines were posted, of which 7173 (16.4%) were from Australia, 18 927 (43.2%) were from Canada and 17 752 (40.5%) were from the UK. This corresponded to an average of 2.57 tweets per user in Australia (range, 1–198), 2.61 tweets per user in Canada (range, 1–433) and 2.62 tweets per user in the UK (range, 1–501).

Expressions of concern

When labelling tweets that expressed concerns, the binary classifier (stage one) achieved a recall of 0.97 and a precision of 0.90. This indicates that the binary classifier missed 3% of tweets that were manually labelled as having expressed a concern, and 10% of tweets labelled as having expressed a concern were manually labelled otherwise. Because the multiclass classifier (at stage two) identified a proportion of these mislabelled tweets in the second round, the overall rate of error in stage one was within 5% of the correct proportion (online supplementary material, Section 4).

The proportion of tweets posted about HPV vaccines from users in the three countries expressing concerns was 18.7% (8215 of 43 852 tweets), but there were differences in these proportions across the three countries (table 2). Canada had the lowest proportion of tweets expressing concerns at 14.9% (2818 of 18 927 tweets), followed by Australia at 19.3% (1388 of 7173 tweets). The UK had the highest proportion of tweets expressing concerns at 22.6% (4009 of 17 752 tweets). Tweets expressing concerns also tended to have smaller audiences compared with tweets not expressing concern about HPV vaccines (online supplementary material, Section 3).

Types of concerns expressed

When identifying concerns related to cues to action, the classifier respectively produced a precision of 0.81 and a recall of 0.74. For perceived barriers, the precision was 0.91, and the recall was 0.92. The classifier was less reliable for the remainder of the concern groups because these types of concerns made up a much smaller proportion, resulting in imbalance in the data, which affects the performance that can be achieved by the classifiers (online supplementary material, Section 5).

Tweets expressing concerns about perceived barriers comprised the largest type of concern by both the proportion of tweets expressing concerns (table 3). The proportions of each group of concerns across the three countries were generally consistent.

Table 3

Number of tweets by country and concern type

Social connections among users

Among users from the three countries who posted about HPV vaccines, 18.2% (3062 of 16 789) were labelled as having expressed concerns (at least half of the tweets about HPV vaccines they posted were labelled as having expressed a concern). The total number of follower connections among the set of 16 789 users was 502 629. Users from the three countries were disproportionately more likely to be followed by users from the same country, creating clusters of users by country (figure 1). Furthermore, users who expressed concerns about the HPV vaccines appear to be more tightly connected within the UK, compared with either Australia or Canada. Figure 1 also highlights that users discussing HPV vaccines in the UK are more often connected to users in Australia and Canada than users in Australia and Canada are connected to each other.

Figure 1

The follower network for Twitter users posting about HPV vaccines is coloured by country (Australia, green; Canada, red; UK, blue). Each node represents a user, and the node sizes are proportional to the number of followers within the user’s network. Nodes are positioned by heuristic to be closer to nodes with which they are better connected, as a way of illustrating the community structure. Darker coloured nodes indicate users for whom at least 50% of their relevant tweets expressed concerns.

To examine the proportion of followers of HPV vaccine tweets, figure 2 examines ‘concern’ and ‘non-concern’ tweets for each of the three countries (to produce six groups represented as circles). Relative to users who did not express concern about HPV, users that did express concerns had a higher proportion of international followers who also expressed concerns (figure 2). Among UK users expressing concerns, 26.1% of followers also expressed concerns (compared with 9.1% of followers among UK users not expressing concerns). Also among UK users expressing concerns, 28.0% of their followers also expressed concerns and were from Australia or Canada, and 9.9% of their followers did not express concerns and were from Australia or Canada. In comparison, among UK users not expressing concerns, only 5.8% of their followers were users not expressing concerns and from Australia or Canada, and 8.3% of their followers were users expressing concerns and from Australia or Canada (online supplementary material, Section 6). This pattern was consistent across each of the three countries. The results indicate that users who mostly expressed concerns were disproportionately well-connected to international users discussing HPV vaccines.

Figure 2

The percentages of followers for all users by expression of concern from Australia, Canada and the UK. The circle represents a concern group of Twitter users, where the circle size is proportional to the number of users. The arrow represents the user following direction. The number represents the percentage of followers, where the number in a circle represents the percentage of followers from the same concern group. Only values above 1.5% are shown (online supplementary material, Section 5) for all values.


This study found that in Australia, Canada and the UK, nearly 1 in 5 of the tweets about HPV vaccines was an expression of concern. Canadian Twitter users less often expressed concerns about HPV vaccines (14.9%) compared with Australia (19.3%) and the UK (22.6%) (table 2). There was a general consistency in the proportions of specific concerns across the three countries, and the most common concerns (46%) were related to ‘perceived barriers’ (ie, logistical challenges and psychological barriers such as perceived harms of receiving the HPV vaccine) (table 3). The results demonstrated that users expressing concerns about HPV vaccines tended to be relatively well-connected to users discussing HPV vaccine concerns in other countries, especially between Canada and the UK.

Previous studies examining the representation of HPV vaccines on Twitter identified slightly higher proportions of negative tweets or tweets expressing concerns, but these studies captured different time periods and did not compare specific countries.52 56 For example, a study of 6 months of Twitter data in the United States (between October 2013 and April 2014) found that 25.1% of tweets were negative.51 Though greater research is required, the balance of positive and negative content appears to vary by source whereby the majority of news content,37 online comments (in response to news articles)42 and tweets have been found to be positive; the majority of YouTube content has been found to be negative.47 In examining the type of concern expressed about the HPV vaccine on other social media sites, researchers have also observed the predominance of perceived barrier (ie, logistical challenges and psychological barriers such as perceived harms of receiving the HPV vaccine).44 45 69 However, while the present research study found that concerns about safety were most common on Twitter, other research found safety to be surpassed or similar in salience to other prevalent themes including conspiracies/search for truth, mistrust for health system and promoting promiscuity.44 45 69 Surian et al analysed topics regarding HPV vaccines on Twitter and found that individuals who posted about ‘harms and conspiracies’ posted more often than other users, suggesting that some users are actively seeking to introduce concerns about HPV vaccines into the public domain.54 The predominance of HPV vaccine concerns about perceived barrier on Twitter indicates the importance of these concerns. It would be valuable to extend this work to examine differences in general vaccine concerns as well as compare concerns towards specific vaccines on Twitter.

International networking on Twitter suggests that vaccine related controversies in one country could reverberate around the world and impact vaccine coverage. Public health professionals and policy-makers must therefore be able to monitor, rapidly identify and react to such concerns (eg, by providing evidence-based responses in real-time and strengthening their own international networks).34 50 70 This research provides public health practitioners and policy-makers with evidence that concerns about ‘perceived barriers’ on Twitter are widespread; effective communication campaigns could be designed and implemented to target this concern in locations where it is likely to have the greatest impact. However, it is important for further research to analyse results by type of sender. It would also be critical for future research to design and evaluate appropriate messaging of such a campaign so that this intervention does not ‘backfire’ and increase hesitancy.71

This study also found that Twitter users expressing HPV vaccine concerns tended to have higher proportions of international connections compared with those not expressing concerns. Given the international connection between Twitter users who express concerns, public health organisations seeking to improve the uptake of HPV vaccines may benefit from tools that help them monitor the impact of vaccine scares on social media locally as well as in other countries in order to pre-empt and respond to misinformation. Greater research would be helpful to further investigate how public health organisations can monitor and intervene to address vaccine concerns. Such support could have been beneficial for Japan and Colombia when the media had a detrimental impact on HPV vaccine coverage.11 30 31

Similar to other studies in this area, our research did not measure whether the expression of concerns on Twitter led to changes in decision-making and coverage. Further research would be beneficial to assess the pathway of HPV vaccine concerns, and whether such concerns have a real-world impact (eg, on vaccine coverage). In particular, future work should consider the relationship between the information about HPV vaccines that enters into the public discourse and the decision-making of individuals and populations. While Canada had the lowest proportion of tweets in which concerns were expressed of the three countries, it also has the lowest rate of HPV vaccine uptake.14 15 18 19 72 Accordingly, we echo Gollust et al’s recent call for greater experimental research designs to make causal assertions about the impact of the media on vaccine coverage.73 Although some studies have begun to do so,36 74 it would be helpful for future research to specifically evaluate the impact of Twitter messages and for moderating variables to also be evaluated.

Together with other studies on the representation of HPV vaccines in the media, our results suggest that it would be useful to monitor early indications of negative influence on attitudes and beliefs on social media. Two studies have independently examined responses on Twitter to specific controversial events including US Representative Michele Bachmann’s claim that HPV vaccines could cause ‘mental retardation’ and Katie Couric’s television segment ‘HPV Vaccine Controversy’ that aired on 4 December 2013.53 75 Mahoney et al evaluated 200 social media posts before and after Bachmann’s comments on the Today Show and found that though most media was positive in tone, compared with Google News, Twitter disseminated more positive HPV vaccine articles and also used more personal accounts as a reference source.53 In contrast, using a random sample of 3595 tweets, Bahk et al found that most sentiment on Twitter towards HPV vaccines before Katie Couric’s episode was negative, and while there was a decrease of negative sentiment immediately after the show aired, negative sentiment returned to baseline after 2 weeks.75 Future research should also investigate how public health organisations should effectively intervene to curb misinformation or ‘fake news’ regarding HPV vaccination.

There were several limitations to this study. First, the findings are specific to Twitter, and while Twitter represents one of the largest populations of social media users, the results are not necessarily representative of the broader public discourse about HPV vaccines in news and online social media.76–79 Twitter is an inherently biased representation of the broader population and is skewed both in terms of age and socioeconomics.80–83 Second, while the location inference method is a standard in the area,84 85 the methods are imperfect.86–89 Third, the study was limited to English-language tweets in three countries, and evaluations of other countries and other languages may have yielded different results. It would be beneficial for future research to expand the focus of analysis to examine diverse countries, as well as conduct more nuanced regional explorations of a single country. Finally, using networks based on which users follow each other does not necessarily capture all of the interactions that occur online. While some argue that interaction with content (liking or retweeting) is a better measure of impact than followers,90 others have argued that many users on Twitter are passive and do not interact with the content,91 and as such, followers may be a better indicator of impact. As this study examined follower networks, it would be helpful for future research to compare followers to different ways of interacting with content in order to better understand the impact of HPV vaccine tweets.


This study characterised the concerns about HPV vaccines expressed by Twitter users in three countries. The UK had the greatest proportion of tweets expressing concerns about HPV vaccines, and Canada had the least, and the types of concerns expressed were relatively consistent across the three countries. Users who expressed concerns about HPV vaccines were generally more closely connected to users in other countries who also expressed concerns, suggesting that controversies and misinformation may be rapidly shared across international boundaries. This research could be used to design public health interventions that address concerns about the HPV vaccine on Twitter. In particular, this study suggests that methods for addressing vaccine concerns may benefit from targeting concerns about perceived barriers to vaccination (including logistical challenges and psychological barriers such as vaccine pain, safety and side effects as a consequence of receiving the HPV vaccine). In addition, further coordination of public health agencies internationally may mitigate vaccine scares.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.
  52. 52.
  53. 53.
  54. 54.
  55. 55.
  56. 56.
  57. 57.
  58. 58.
  59. 59.
  60. 60.
  61. 61.
  62. 62.
  63. 63.
  64. 64.
  65. 65.
  66. 66.
  67. 67.
  68. 68.
  69. 69.
  70. 70.
  71. 71.
  72. 72.
  73. 73.
  74. 74.
  75. 75.
  76. 76.
  77. 77.
  78. 78.
  79. 79.
  80. 80.
  81. 81.
  82. 82.
  83. 83.
  84. 84.
  85. 85.
  86. 86.
  87. 87.
  88. 88.
  89. 89.
  90. 90.
  91. 91.
View Abstract


  • Contributors GKS, AGD and MK conceptualised and designed the study. DS and AGD acquired the data. GKS drafted the manuscript. All authors analysed and interpreted the data, critically revised the manuscript for important intellectual content and approved the final version.

  • Funding GKS was supported by the CIHR Vanier Canada Graduate Scholarship and Queen Elizabeth II Diamond Jubilee Scholarship programmes. AD is an investigator on a related NHMRC Project Grant (APP1128968). The funding source had no role in the design of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript and the decision to submit the manuscript for publication.

  • Competing interests None declared.

  • Ethics approval Macquarie University Human Research Ethics Committee (number 5201401028) and the University of Melbourne’s Research Ethics Board (number 1647488.1).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional data are available. Due to ethical approvals and the Twitter Terms of Service, the individual identifiers for Twitter posts and users cannot be provided with their manual or automatic labels.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.