You are here

  • Home
  • Archive
  • Volume 2, Issue 2
  • Do adverts increase the probability of finding online cognitive behavioural therapy for depression? Cross-sectional study

Article Text

Article menu
XML
Health informatics
Research
Do adverts increase the probability of finding online cognitive behavioural therapy for depression? Cross-sectional study
  1. Ray B Jones1,
  2. Lesley Goldsmith1,
  3. Paul Hewson2,
  4. Maged N Kamel Boulos1,
  5. Christopher J Williams3
  1. 1School of Nursing and Midwifery, Plymouth University, Plymouth, UK
  2. 2School of Computing and Mathematics, Plymouth University, Plymouth, UK
  3. 3Institute of Health and Wellbeing, MVLS University of Glasgow, Glasgow, UK
  1. Correspondence to Professor Ray B Jones; ray.jones{at}plymouth.ac.uk

Abstract

Objective To estimate the effect of online adverts on the probability of finding online cognitive behavioural therapy (CBT) for depression.

Design Exploratory online cross-sectional study of search experience of people in the UK with depression in 2011. (1) The authors identified the search terms over 6 months entered by users who subsequently clicked on the advert for online help for depression. (2) A panel of volunteers across the UK recorded websites presented by normal Google search for the term ‘depression’. (iii) The authors examined these websites to estimate probabilities of knowledgeable and naive internet users finding online CBT and the improved probability by addition of a Google advert.

Participants (1) 3868 internet users entering search terms related to depression into Google. (2) Panel, recruited online, of 12 UK participants with an interest in depression.

Main outcome measures Probability of finding online CBT for depression with/without an advert.

Results The 3868 users entered 1748 different search terms but the single keyword ‘depression’ resulted in two-thirds of the presentations of, and over half the ‘clicks’ on, the advert. In total, 14 different websites were presented to our panel in the first page of Google results for ‘depression’. Four of the 14 websites had links enabling access to online CBT in three clicks for knowledgeable users. Extending this approach to the 10 most frequent search terms, the authors estimated probabilities of finding online CBT as 0.29 for knowledgeable users and 0.006 for naive users, making it unlikely CBT would be found. Adding adverts that linked directly to online CBT increased the probabilities to 0.31 (knowledgeable) and 0.02 (naive).

Conclusions In this case, online CBT was not easy to find and online adverts substantially increased the chance for naive users. Others could use this approach to explore additional impact before committing to long-term Google AdWords advertising budgets.

Trial registration This exploratory case study was a substudy within a cluster randomised trial, registered on http://www.clinicaltrials.gov (reference: NCT01469689). (The trial will be reported subsequently).

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

https://doi.org/10.1136/bmjopen-2011-000800

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Article summary

    Article focus

    • In 2011 in the UK, what is the chance of finding online CBT for depression?

    • Does online advertising increase that chance?

    Key messages

    • Online CBT for depression is recommended by the National Institute for Health and Clinical Excellence for people with depression, but we did not know how easy it is to find or if online adverts help find it.

    • People with depression are unlikely to find online CBT by chance: the probabilities of finding online CBT using Google was 0.29 for knowledgeable users and 0.006 for naive users. Adding adverts that linked directly to online CBT increased the probabilities to 0.31 (knowledgeable) and 0.02 (naive).

    • This method of assessing probability of finding topics should be used before committing to long-term online advertising.

    Strengths and limitations of this study

    • The results are likely to be typical for anyone with depression in the UK in 2011. The approach of identifying typical search terms and simulating knowledgeable and naive user experience, with and without adverts, is generalisable to other conditions and populations.

    • Search results will change over time.

    Background

    Less than 60% of people with diagnosable depression or anxiety seek formal help from practitioners; this represents a significant treatment gap.1 The remainder may access informal care and support, alternative therapies, private arrangements such as counselling, use of the voluntary sector and the internet and other sources of information.

    Nearly one in five British internet users search for information related to mental health.2 Among younger people, this may be much higher. Reasons cited included anonymity, the amount of information available and easy access.3–5 The level of searching for information on depression online has increased. In 2000, depression was not in the 10 most frequently searched for topics, but by 2005, it was the fifth most common topic.5

    Others have investigated quality of health information websites, including those for depression,6 and attempted to rate the quality of these websites.7–9 Patients searching for health information online may be unable to find what they are looking for,10 possibly due to suboptimal search strategies.11 Most people follow-up between the first three and 10 results.3 12 We have not found studies of how users seek online help for depression but most people searching for health information start with a search engine.10 Google's share of the search market is debatable13 14 but we can assume that a majority in the UK searching for health information will use Google. Search results may now be personalised according to user's location15 and previous searches.16 17 Some regard this personalisation as intrusive and this has led to the development of software to hide user characteristics.18 Search results will also change over time because of adjustment of Google's databases to the changing pattern of websites. This means that there is no standard response to a Google search; however, we can ‘average’ the experience of several users to explore ease of finding given topics at any time.

    Although not everyone searching online using terms such as depression is depressed, we need to make sure that the best support is available for those who are. Online cognitive behavioural therapy (CBT) is self-help treatment recommended by the National Institute for Health and Clinical Excellence (NICE) in the UK for mild to moderate depression.19 The range of resources includes licenced (paid for) sites such as Beating the Blues20 and free access websites providing access to CBT life skills resources (eg, Living Life to the Full (LLTTF)21 and MoodGYM).22

    Given the NICE recommendation and the predominance of Google, it would be reasonable to expect websites delivering access to online CBT to have relatively high profiles and be easily visible to users on a Google search. We were struck by the large geographical variation within the UK in the uptake of LLTTF. The highest rate (in Kirkwall, Scotland) was 15 times that of the lowest (Wigan, England). Such variation could not be explained by variation in prevalence of depression and was most likely explained by lack of awareness of online therapies for depression. We have been exploring ways of increasing awareness.23 One method of doing this was to use online advertising, in particular, Google AdWords.24 25

    Those paying for an AdWords campaign set up one or more adverts (see figure 1 for example) and enter keywords to help determine when the advert is shown. AdWords determine if an advert is relevant to a given user's search based on (in AdWords terminology) exact matches, broad matches and phrase matches with the user's entered search term.26 AdWords displays adverts as a sponsored link, either at the top of the list of search results or in the right-hand panel, depending on the phrase entered, the price you offer per advert and bids from competing adverts, and (if requested) by estimated location of the user. Google AdWords may be embedded in searches in other websites (the Google Display Network).27

    However, diverting users who would have found CCBT sites without the aid of adverts, to finding it via an advert, is a waste of resources. In this case study, we reviewed the likely ‘search experience’ using Google of people searching using the term ‘depression’ to estimate the chance of finding online CBT and the improved probability by adding an online advert.

    Methods

    Ethics and trial registration

    This was an exploratory substudy within a pilot cluster randomised trial. The whole study was approved by the Plymouth NHS Research Ethics Committee (reference. 11/H0203/8), is registered on http://www.ClinicalTrials.gov (reference: NCT01469689) and will be reported subsequently.

    Setting and population

    We reviewed search terms entered by 3868 internet users in the UK who used the Google search engine for terms related to depression and who clicked our presented Google advert (figure 1) between 17 April 2011 and 31 August 2011.

    We enabled 126 keywords in our AdWords campaign and allowed exact matches, broad matches and phrase matches. Those clicking on the advert were taken to a website which asked them to confirm their location (postcode area) and to complete a widely used depression rating questionnaire (PHQ928) and then offered links to four sites addressing low mood in different ways. Our main objective was to offer a link to online CBT, in particular, LLTTF and MoodGYM.29 The NHS research ethics committee that considered our proposal asked us to add a link to Samaritans.30 (Samaritans is a secular UK charity aimed at providing emotional support to anyone in emotional distress or at risk of suicide mainly through their 24/7 confidential telephone helpline.) To keep symmetry in the exit page of the website, we added NHS Choices (non-CBT information about depression).31 The order of links presented on the page was randomised within each row, the top row always being the two CBT sites and the bottom row Samaritans and NHS Choices.

    Search terms used

    We reviewed the presentation of our advert and the search terms entered by users (who subsequently clicked on our advert) to understand better what terms people may use and how many may search directly for online CBT.

    Exploring the results of normal Google search

    Given that the results of Google searches are to some degree personalised and therefore will vary by computer used, to assess which websites would be found by users, we recruited a user panel of 12 people. We contacted a sample of previous users of LLTTF and through an AdWord's advert (using the keyword depression) sought volunteers in our target areas (12 of 121 postcode areas in Britain).23 These people, in a range of locations, are likely to have had search history including previous searches for depression and so would be presented with different Google search results. These results would be representative of those obtained by people in our target areas searching for depression-related terms. Although we tried to target our study postcode areas in recruiting the panel, there was some ‘leakage’. Nine of 12 were in target areas (Darlington, Lancaster, Orkneys, London SW, Kingston and Liverpool) and three were in other areas (Twickenham, Gloucester and Dorchester). We asked our user panel to search Google using the single word ‘depression’ and email us a screen shot of the first page of results (usually 10 results). We also wanted to explore the variability of search results by different entered terms. We felt unable to ask our panel to carry out and report more than one search so two of the authors searched for the next nine most entered search terms (table 2).

    Estimating the chance of finding online CBT from websites

    We assessed the websites found by people using the term ‘depression’ to see how easy it was to find online CBT. We first assumed a knowledgeable user and assessed if online CBT (in this case specifically LLTTF or MoodGYM) could be reached in three clicks by someone who knew that online CBT sites existed. For those websites where online CBT could be reached in three clicks, we calculated the probability that someone (ie, an ‘extreme’ novice) who made completely random choices would find online CBT purely by chance. To do this, we noted how many choices were involved at each level of the shortest pathway to online CBT to calculate an approximate probability of reaching online CBT.

    Model to estimate probabilities of finding online CBT

    We used a number of assumptions (see results) to estimate the probability of finding online CBT with and without Google adverts. Figure 2 shows a schematic of the samples used and how information from different stages was used to calculate the probabilities.

    Figure 2
    Figure 2

    Schematic of methods. CBT, cognitive behavioural therapy.

    Results

    Search terms

    By 31 August 2011, Google had presented our advert 299 369 times to various Google searches that included combinations of our keywords, resulting in 3868 ‘clicks’. Google reported the 1748 most frequently entered search terms entered by users which were responsible for 71 124 (24%) of the presentations. Examples are shown in table 1. Only four (0.2%) search terms of 1748 included either the words cognitive, behavioural or behavioural and this resulted in 72 presentations (one in 1000) of our advert.

    View this table:
    Table 1

    Examples of search terms as entered (including spelling mistakes and typing errors) by users for which our Google Advert was displayed and for which the user clicked on the advert

    Ordering the search terms by the number of resultant clicks on our advert, we selected the 10 top terms (table 2). Although these 10 terms only resulted in 1407 (36%) of the clicks, other entered terms, including the terms in table 2 with additional words or misspelled, were likely to have resulted in similar Google search results.

    View this table:
    Table 2

    The top 10 entered search terms showing the number of resultant clicks on our advert

    AdWords matched the entries made by users with the 126 keywords using broad matches, exact matches and phrase matches. We reviewed the most popular keywords as listed by AdWords and the full search terms that were entered by users. ‘Depression’ as a single keyword in our Google AdWords campaign resulted in two-thirds of the advert presentations and over half of the clicks.

    Websites offered by Google search

    Given that Google search results are to some degree personalised by location and previous searches, we wanted to know what a ‘typical’ search result would be. As the single word ‘depression’ was the most frequent search, we asked our UK user panel to search with that single word. The variability of search results using the single term depression was relatively small; only 14 sites in total were presented to the 12 users (table 3). (If searches were exactly the same, there would only be 10 sites, but if searches were all completely different, there would be 120.) Six sites were seen by all 12 users. The NHS Choices appeared in the top three for all.

    View this table:
    Table 3

    Websites (with hyperlinks) identified by 12 participants searching Google using the term ‘depression’: showing the number of users for whom the website was on first page of results

    Finding online CBT from Google displayed websites

    One researcher (LG) explored each of the 14 websites in table 3 to see if online CBT could be found within three clicks. Four of the website had links enabling access to either MoodGYM or LLTTF in three clicks (table 3).

    We then estimated the probability of finding online CBT on these four websites by chance, that is, for an ‘extreme naive’ user. Table 4 is constructed knowing pathways that will lead to online CBT and while following that path, counting the total number of choices available. For example, on the Royal College of Psychiatrists site, the ‘landing’ depression web page from Google currently has seven main choices within the body of the text (figure 3). It also has some 49 items on the side menu bar, 12 on the tool bar menu, seven to 10 additional information items, 16 links on references, three links at the bottom of the page on leaflets, an online questionnaire and eight links to the ‘small print’.

    View this table:
    Table 4

    Estimated probabilities of a naive user randomly finding online cognitive behavioural therapy (CBT) from four websites where a knowledgeable user could find CCBT in three clicks

    Figure 3
    Figure 3

    Screen shot from Royal College of Psychiatrists ‘depression’ landing page, showing (added labels) side bar menu, tool bar menu and additional information.

    We assumed that our naive user with depression having arrived at this page from Google is able to avoid the distractions of the peripheral links and just choose one of the seven from the body of the text (figure 4) and then find ‘Free Online CBT resources’ which is one of 20 choices on the second screen (figure 5).

    Figure 4
    Figure 4

    Screen shot from Royal College of Psychiatrists ‘depression’ landing page, showing link words in the body of the text and other headings that were not linked.

    Figure 5
    Figure 5

    Section from second screen for http://www.RCPsych.ac.uk that has route to CCBT.

    Other search terms

    Examination of the first page of 10 results for each of the next nine search terms resulted in over 60 different web addresses from 37 organisations. (With 2 researchers, 9 search terms and 10 results each, the total could be 2×9×10=180.) Five of the websites could lead to online CBT, and these were among the 10 most frequently presented (table 5). Four of the five were those displayed for a search on the single word ‘depression’ (table 3).

    View this table:
    Table 5

    The 12 most frequently presented websites (out of 37 in total) from searches by two researchers using nine different search terms, showing the number of times presented (out of 180 (=2×9×10) search results) and whether the site can be used to find online cognitive behavioural therapy (CBT) in three clicks

    The Samaritans websites (including main website and local variations) were only displayed on entering the Samaritans search term. (Samaritans as a search term was picked up and displayed for our advert as a result of our website having Samaritans as one of the destination links.) In summary, table 6 shows the likely mean number of websites among the first 10 that will link to online CBT in three clicks for a knowledgeable user.

    View this table:
    Table 6

    Search terms and the mean number of Google search results, out of 10 on the first page, that would offer access to online cognitive behavioural therapy (CBT) in three clicks

    Probability model

    What, therefore, is the overall probability of finding an online CBT website by searching for ‘depression’? How does this vary between the optimistic scenario of a knowledgeable user and a more pessimistic scenario of naive user making random choices? We need to make a number of assumptions, namely that:

    1. We can (for the time being) exclude Google Ads from the choices.

    2. Users enter one of 10 search terms, in proportion to the number of clicks as shown in table 2.

    3. Users will be offered 10 sites on the first page by Google search.

    4. The number of sites from which online CBT can be found by a knowledgeable user is as shown in table 5.

    5. Google results will comprise those shown in table 3 for the term ‘depression’ and those shown in table 5 for the next nine search terms.

    6. Knowledgeable users have probability of 1 of finding online CBT from the five websites shown in table 5 and probability of 0 from other sites.

    7. Naive users will have a probability of finding online CBT from those sites calculated by the number of options at each level (as demonstrated in table 4) and probability of 0 for other websites.

    The probabilities of finding online CBT are summarised in table 7. A knowledgeable user has a one in three chance, but a naive user has chance of just 1/150. These are probably overestimates as the search results for other entered search terms are unlikely to present more websites leading to online CBT.

    View this table:
    Table 7

    Probabilities of finding online cognitive behavioural therapy (CBT) for knowledgeable and naive users depending on search term entered

    Change in probabilities through addition of Google advert

    What happens to these probabilities if we take into account Google Ads? We have no data on the relative chance of someone clicking on a ‘paid for’ Google Ad rather than a Google search result. However, we can estimate from our own Google Ads campaign by using the ‘click through rate’. This refers to the number of times the advert is presented at which a user chooses the advert. Our click through rate was 1.5%. The number of adverts displayed by Google, either on the right side of the search panel or as the first few searches, in the sponsored links areas is variable. For our users searching for depression, there were typically eight adverts. If we assume that a typical search and adverts screen therefore displays 10 search results and eight adverts, the user has an overall choice of 18 items. We might estimate that the chance of clicking on any one advert is 1.5% so the chance of clicking on an advert is 8×1.5%=12% and the chance of clicking on a Google search result is 88%.

    Box 1 shows the Google Ads that were seen by two or more of our users, one of which (The Royal College of Psychiatrists—RCPsych) has a one in 140 chance of reaching online CBT as shown in table 4. If we assume therefore that in addition to the 10 Google searches each user is presented with eight Ads, one of which is http://www.rcpsych.ac.uk, then the probabilities of reaching online CBT, using figures from table 7, becomeKnowledgeable user:(0.88×0.33)+(0.12×1/8×1/140)=0.2904+0.00011=0.2905Naive user:(0.88×0.0069)+(0.12×1/8×1/140)=0.0061+0.00011=0.0062.

    Box 1

    Websites advertised by Google AdWords seen by two or more from the panel of 12 participants

    We can now calculate the increase in probability of adding an advert that leads directly to online CBT and assume that this is one of the eight adverts showing alongside that for the RCPsych.

    The probability of finding online CBT via an advert becomes(0.12×1/8×(1/140+1))=0.0151.

    So probabilities of finding online CBT becomeKnowledgeable user:0.2904+0.0151=0.3055Naive user:0.0061+0.0151=0.0212.

    So adding a Google AdWords advert which leads directly to online CBT would improve the chance of finding CCBT for knowledgeable users a little (from 29% to 31%) but would improve the chances of a naive user from one in 167 to one in 66.

    Discussion

    Increasingly in UK and elsewhere, people are encouraged to self-refer for support and guidance in using online CBT. For example, Increasing Access to Psychological Therapies (IAPT)—a major Department of Health initiative with delivery across England—is promoting self-referral and finding positive benefits.32 This is also seen in the national telephone support line NHS Living Life. Approaches such as these are likely to access a range of people who might not otherwise seek help via their GP.33 Support and guidance are recommended for all forms of CBT resources—including online CBT.34 Although national treatment guidelines such as NICE (2009) and SIGN (2010)19 35 recommend the addition of support, many people accessing online CBT through free sites will do so without support.

    This study showed that the 3868 people who clicked on our advert typically searched with the single word depression, although there were a great variety of terms entered, including misspellings. However, our study demonstrated that even a knowledgeable person may have to search quite hard to find online CBT, and it would be difficult, or extremely unlikely, for a naive user of depression websites (ie, someone not specifically searching for online CBT for depression and knowing how to find it) to find online CBT websites easily using a Google search. Although our estimates use a large number of assumptions, we consider that a nearly threefold increase in the probability of finding online CBT by the addition of a Google Advert is likely to lead to more people with depression registering and using online CBT. The real test of this will be if there is an increase in uptake demonstrated in our cluster randomised trial comparing intervention with control areas.

    Based on our ‘click through rate’ (the number of times users clicked on our advert divided by the number of times it was presented) of 1.5% and that typically eight adverts are displayed on a page of search results, we estimated that users may click on an advert in 12% of cases when Google displayed search results and adverts. Although this is based on our empirical data, online adverts may not be viewed positively by internet users who are looking for credible information on the web. AdWords are sometimes used to promote bogus, misleading (eg, ‘miracle cures’) and malware-ridden sites, and many users will avoid them. Our own advert (figure 1), mentioning NHS and with a ‘.org.uk’ URL, may have had a better click through rate than other online adverts,36 so we may have overestimated the impact of online adverts.

    The Department of Health was heavily criticised for spending £2.5 million on AdWords between February 2009 and January 2010.37 The Department of Health declined to give further details of how they had spent this money or its cost–effectiveness. By judicious design of the website, the NHS Choices and other NHS websites should bring high profile search results. In this study, for example, the NHS Choices depression section was the only website that regularly appeared in the top three search results. If people find a website via a normal web search, then online advertising such as AdWords may be a waste of money. However, this study shows that, if there is substantial ‘competition’ for a user's attention, advertising may be worthwhile. For example, the Royal College of Psychiatrists seems to be spending quite large sums on Google AdWords as their advert appears frequently alongside their website being found by ‘normal’ Google search. For people with depression, given that adverts will in any case appear, having an advert for RCPsych that gives the user a one in 140 chance of finding online CBT for depression improves their chances very slightly.

    This exploration of how choices may be made given the websites and adverts currently being found by Google gives some indication that it would be worthwhile. The number of routes that could be taken, even on websites from which it is possible to reach online CBT, also shows the bewildering choice facing those with depression using the internet.

    Another approach to assessing search experience is through observation in a laboratory3 11 setting or even ‘in the wild’, but recruiting, meeting and observing people with depression while they search the internet presents many ethical and practical difficulties. We were able to recruit 12 anonymous users who are likely to have searched for depression before, at various locations around the UK. They were able to supply us with information of their searches from their natural ‘habitat’, although we restricted their searching to just one search term. Ideally, we would ask the panel to search for more search terms but we thought this would jeopardise getting any response. The results suggested that although there was some modification of search results by users' previous search history and location, the variability was not that great. Although our sample of 12 users is small, given the lack of variation, it seems unlikely that a bigger sample would have identified other websites that would have increased the probability of finding online CBT.

    Our study is also limited in that we only considered Google Search and not, for example, Google Search or Google AdWords embedded in partner sites, blogs or YouTube videos (the so-called Google Display Network). Also, not only do websites change but also Google search results change by location, user and over time. Website owners are also constantly modifying their websites and taking steps to raise their profiles for search engines such as Google. Google is also not the only way of advertising online and others have used, for example, paid location and demographic targeted adverts in Facebook on.38

    The approach we have taken to assess the ease of finding CCBT could easily be replicated in future. The general idea of assessing the probabilities that expert and naive users may have in finding a particular website is applicable to anyone considering the addition of online adverts to promote online services or resources. Online adverts may compete both with other adverts as well as with the results of organic search, and health service providers considering the use of online adverts need a strategy that calculates the added (marginal) benefit and cost.

    Conclusions

    More work is needed into how people make choices when confronted by different Google search results and adverts. Users of Google AdWords would also be advised to consider the context in which they are advertising and not just the data reported by Google Analytics. The results of a search and the additional impact of advertising should be explored in more detail before committing to a long-term Google AdWords advertising budget promoting access to healthcare resources.

    Acknowledgments

    We would like to thank Ian Mayer of Mirata Ltd for help in extracting data from LLTTF and Robert Stillwell for developing the project website. We would like to thank the Royal College of Psychiatrists who have given permission for the inclusion of screenshots from their website.

    References

    1. ↵
      1. Kohn R,
      2. Saxena S,
      3. Levav I,
      4. et al
      . The treatment gap in mental health care. Bull World Health Organ 2004;82:858–66.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Powell J,
      2. Clarke A
      . Internet information-seeking in mental health: population survey. Br J Psychiatry 2006;189:273–7.
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Buhi ER,
      2. Daley EM,
      3. Fuhrmann HJ,
      4. et al
      . An observational study of how young people search for online sexual health information. J Am Coll Health 2009;58:101–11.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Horgan Á,
      2. Sweeney J
      . Young students' use of the Internet for mental health information and support. J Psychiatr Ment Health Nurs 2010;17:117–23.
      OpenUrlCrossRefPubMed
    5. ↵
      1. Sillence E,
      2. Briggs P,
      3. Harris PR,
      4. et al
      . Going online for health advice: changes in usage and trust practices over the last five years. Interact Comput 2007;19:397–406.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Ferreira-Lay P,
      2. Miller S
      . The quality of internet information on depression for lay people. Psychiatr Bull 2008;32:170–3.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Breckons M,
      2. Jones R,
      3. Morris J,
      4. et al
      . What do evaluation instruments tell us about the quality of complementary medicine information on the internet? J Med Internet Res 2008;10:e3.
      OpenUrlCrossRefPubMed
    8. ↵
      1. Eysenbach G,
      2. Powell J,
      3. Kuss O,
      4. et al
      . Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287:2691–700.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Eysenbach G,
      2. Thomson M
      . The FA4CT algorithm: a new model and tool for consumers to assess and filter health information on the internet. In: Kuhn KA, Warren JR, Leong TY, eds. Medinfo 2007: Proceedings of the 12th World Congress on Health. Amsterdam: I O S Press, 2007:142–6.
    10. ↵
      1. Fox S
      . Pew Internet and American Life Project. Online Health Search, 2006. http://www.pewinternet.org/∼/media//Files/Reports/2006/PIP_Online_Health_2006.pdf.pdf (accessed 8 Nov 2011).
    11. ↵
      1. Eysenbach G,
      2. Köhler C
      . How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002;324:573–7.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Granka L,
      2. Joachims T,
      3. Gay G
      . Eye-tracking analysis of user behavior in www search. ACM Conference on Research and Development in Information Retrieval. 2004. http://dl.acm.org/citation.cfm?id=1009079 (accessed 28 Feb 2012).
    13. ↵
      Netmarketshare. Search Engine Market Share. http://marketshare.hitslink.com/search-engine-market-share.aspx?spider=1&qprid=4 (accessed 27 Feb 2012).
    14. ↵
      1. Schmidt E
      . Testimony Before the Senate Committee on the Judiciary Subcommittee on Antitrust, Competition Policy, and Consumer Rights. 2011. https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B5JQZrEEQaEONDJkZWI1MzUtMzk5Mi00ZDRhLWIyZmMtMWRkOWU1MmU5ZmZk&hl=en_US (accessed 28 Feb 2012).
    15. ↵
      Google Web Search Help: Location. http://support.google.com/websearch/bin/answer.py?hl=en&answer=179386 (accessed 27 Feb 2012).
    16. ↵
      1. Smyth B,
      2. Coyle M,
      3. Briggs P
      . Communities, Collaboration, and Recommender Systems in personalized web search. In: Ricci F, Rokach L, Shapira B, et al., eds. Recommender Systems Handbook. New York: Springer US, 2011:579–614.
    17. ↵
      Google Support. Basics: Search History Personalization. http://support.google.com/accounts/bin/answer.py?hl=en&answer=54041 (accessed 27 Feb 2012).
    18. ↵
      1. Toubiana V,
      2. Subramanian L,
      3. Nissenbaum H
      . TrackMeNot: Enhancing the Privacy of Web Search. 2011. http://arxiv.org/abs/1109.4677 (accessed 27 Feb 2012).
    19. ↵
      National Institute for Clinical Excellence. Depression: The Treatment and Management of Depression in Adults (Update). 2009. http://www.nice.org.uk/guidance/CG90/NICEGuidance (accessed 12 Dec 2011).
    20. ↵
      Beating the Blues. http://www.beatingtheblues.co.uk/patients/ (accessed 19 Feb 2012).
    21. ↵
      1. Williams C
      . Living Life to the Full. http://www.livinglifetothefull.org.uk (accessed 28 Feb 2012).
    22. ↵
      The Australian National University. MoodGYM. http://moodgym.anu.edu.au/welcome (accessed 27 Feb 2012).
    23. ↵
      1. Jones R,
      2. Goldsmith L,
      3. Williams C
      . Raising Awareness of Online Therapies for Depression: Pilot Study. 2010. http://www.bupafoundation.co.uk/Annual-Review/Ray-Jones.htm (accessed 27 Feb 2012).
    24. ↵
      1. Estrada CA,
      2. Krishnamoorthy P,
      3. Smith A,
      4. et al
      . Marketing to increase participation in a web-based continuing medical education cultural competence curriculum. J Contin Educ Health Prof 2011;31:21–7.
      OpenUrlPubMed
    25. ↵
      1. Ramo DE,
      2. Hall SM,
      3. Prochaska JJ
      . Reaching young adult smokers through the internet: comparison of three recruitment mechanisms. Nicotine Tob Res 2010;12:768–75.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      Google AdWords Help. http://support.google.com/adwords/bin/answer.py?hl=en&answer=6100 (accessed 27 Feb 2012).
    27. ↵
      Google Display Network. http://www.google.com/ads/displaynetwork/ (accessed 27 Feb 2012).
    28. ↵
      1. Kroenke K,
      2. Spitzer RL
      . The PHQ9: a new depression diagnostic and severity measure. Psychiatr Ann 2002;32:509–21.
      OpenUrlCrossRefWeb of Science
    29. ↵
      1. Christensen H,
      2. Griffiths KM,
      3. Jorm AF
      . Delivering interventions for depression by using the internet: randomised controlled trial. BMJ 2004;328:265.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      Samaritans. http://www.samaritans.org/ (accessed 14 Dec 2011).
    31. ↵
      NHS Choices. NHS Choices, Your Health, Your Choices. Depression. http://www.nhs.uk/conditions/depression/Pages/Introduction.aspx?WT.mc_id=61006 (accessed 12 Dec 2011).
    32. ↵
      1. Brown JS,
      2. Boardman J,
      3. Whittinger N,
      4. et al
      . Can a self-referral system help improve access to psychological treatments? Br J Gen Pract 2010;60:365–71.
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Jorm AF,
      2. Griffiths KM,
      3. Christensen H,
      4. et al
      . Actions taken to cope with depression at different levels of severity: a community survey. Psychol Med 2004;34:293–9.
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Gellatly J,
      2. Bower P,
      3. Hennessy S,
      4. et al
      . What makes self-help interventions effective in the management of depressive symptoms? Meta-analysis and meta-regression. Psychol Med 2007;37:1217–28.
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      SIGN. Guideline 114: Non-Pharmaceutical Treatment of Depression in Adults: A National Clinical Guideline. 2010. http://www.sign.ac.uk/pdf/sign114.pdf (accessed 19 May 2011).
    36. ↵
      Wikipedia. Clickthrough Rate. 2012. http://en.wikipedia.org/w/index.php?title=Clickthrough_rate&oldid=474350612 (accessed 27 Feb 2012).
    37. ↵
      publictechnology.net. NHS Online Google Ads Come Under Fire. http://www.publictechnology.net/sector/nhs-health/nhs-online-google-ads-come-under-fire (accessed 20 Oct 2011).
    38. ↵
      1. Fenner Y,
      2. Garland SM,
      3. Moore EE,
      4. et al
      . Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res 2012;14:e20.
      OpenUrlCrossRefPubMed
    View Abstract

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • To cite: Jones RB, Goldsmith L, Hewson P, et al. Do adverts increase the probability of finding online cognitive behavioural therapy for depression? Cross-sectional study. BMJ Open 2012;2:e000800. doi:10.1136/bmjopen-2011-000800

    • Contributors RBJ had the idea for the study, is principal investigator, grant holder, was responsible for day-to-day management of the project, carried out most analyses, wrote and edited the paper. LG contributed to the research proposal, is co-grant holder, undertook day-to-day management, did some of the analyses, helped write and edit the paper. CJW contributed to the research proposal, is co-grant holder and edited the paper. PH advised on the interpretation of the data and edited the paper. MNKB advised on the interpretation of the data, co-wrote and edited the paper.

    • Funding The project was funded by a grant from the BUPA Foundation Philip Poole-Wilson Seedcorn Fund. All authors are independent from the funding body.

    • Competing interests CJW is the designer and author of the LLTTF site.

    • Ethics approval Ethics approval was provided by South West Ethics Committee. Ethics committee approval reference: 11-H0203-8.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No other data are available to share.