Background

HIV infection is a major public health concern across Europe, and transmission during sex between men is increasingly common. Among the 48 European countries consistently reporting HIV diagnoses and mode of transmission over the period 2004 to 2010, the number of new HIV diagnoses among men having sex with men (MSM) increased by 42 %, from 7,621 in 2004 to 10,854 in 2010. (European Centre for Disease Prevention and Control/WHO Regional Office for Europe: HIV/AIDS surveillance in Europe 2010).

Robust information on the values, prevention needs and behaviours of stigmatised minority groups is difficult to gather. Defining groups like ‘men who have sex with men’ make estimating the size of the ‘true’ population difficult and the absence of sampling frames mean random sampling is challenging, if not impossible. Therefore, most studies of MSM have relied on convenience samples recruited in community venues (saunas, events, bars) or clinical services (STI or HIV clinics). In recent years, the Internet has been an important setting for recruiting larger samples of MSM that are more diverse in terms of age, education, bisexuality and geographic distribution than paper surveys recruited through gay community settings (Ross et al. 2000; Whittier et al. 2004; Elford et al. 2004). In countries with widespread Internet access, Internet-recruited MSM samples have been shown to approximate the regional distribution of MSM (Marcus et al. 2009a, b).

The European Centre for Disease Prevention and Control (ECDC) has an objective to strengthen the capacity of European Union (EU) countries to prevent and control infectious diseases, with HIV and other sexually transmitted infections (STIs) among the priority diseases. Monitoring the various European HIV and STI epidemics is vital to the corresponding public health responses. Since 2008, the ECDC and the World Health Organization’s Regional Office for Europe have jointly been carrying out HIV/AIDS surveillance in Europe (European Centre for Disease Prevention and Control. Mapping of HIV/STI Behavioural Surveillance in Europe. Stockholm 2009). ECDC is mandated to strengthen and coordinate second-generation surveillance which includes behavioural as well as biological measures. EMIS (European MSM Internet Survey, see www.emis-project.eu) moves towards third-generation surveillance by incorporating HIV prevention needs. We define prevention needs as the psychosocial and material requirements for control over HIV risk and precaution behaviours. They are dictated by the theories of behaviour being employed but may include such concepts as knowledge, social norms, control beliefs, access to resources, motor and interpersonal skills.

In 2008, a study of HIV/STI behavioural surveillance in countries of the EU and European Free Trade Association mapped the current state of HIV/STI-related behavioural surveillance across Europe, with a focus on vulnerable groups such as MSM (European Centre for Disease Prevention and Control. Mapping of HIV/STI Behavioural Surveillance in Europe. Stockholm 2009). Of the 31 countries surveyed, 28 responded, of which 16 claimed an established behavioural surveillance system, 14 of which expressly included MSM. Another four countries carried out periodic behavioural surveys among MSM. The study acknowledged that in some small countries such systems may be difficult to justify, and in others, MSM may be especially hard to recruit due to social, cultural or religious barriers. When sufficiently contextualised, regular and comparable behavioural surveillance can help to improve our understanding of trends in diseases and allow more precise planning and evaluation of prevention responses (Brown 2003; Garnett et al. 2006; McGarrigle et al. 2006).

To date, there has been little collaboration across EU countries in data collected from MSM. Comparability is hampered by partial lack of data, different questions and response sets, different recruitment methods and different biological surveillance and health care systems. Moreover, different ways of accessing and defining MSM lead to different sample compositions, and different questions targeting the same concept may result in answers that are not comparable.

EMIS is a multi-language, pan-European, cross-sectional, HIV prevention needs assessment for MSM, including both measures of risk behaviours and of unmet HIV prevention needs. The survey questions were designed to allow maximum comparisons with past and future national and regional surveys, and to allow the construction of indicators of national HIV responses suggested by UNAIDS (UNAIDS: Monitoring the Declaration of Commitment on HIV/AIDS: Guidelines on Construction of Core Indicators, Geneva 2009) and ECDC (European Centre for Disease Prevention and Control. Mapping of HIV/STI Behavioural Surveillance in Europe. Stockholm 2009) EMIS sought to advance the harmonisation of survey methods and questions and to generate comparable data between countries. It provides data for the planning of interventions and to facilitate the monitoring of changes over time in HIV-related behaviours, needs and interventions among MSM. If repeated, it can serve as the first wave of a pan-European, third-generation HIV surveillance system that encompasses data about prevention needs as well as behavioural risks.

There is much to be learnt by country comparisons of HIV epidemics. Although a few international collaborations in MSM community research have been attempted, none has delivered MSM samples of a reasonable size in a large number of nations. EMIS did so and are describing its research processes in detail to allow other research teams to identify elements of difference to their processes. We draw readers’ attention in particular to the extensive engagement with community groups and members in both the research agenda and the acceptability of the products.

Methods and Design

Aims and Objectives

In early discussions between the EMIS Associated Partners, the prevention-planning objectives of EMIS were to identify prevention needs commonly unmet across diverse groups of MSM (priority aims), and to identify subgroups of MSM who have multiple prevention needs poorly met (priority target groups). Other objectives included: capacity building and knowledge transfer for European online research among MSM; the generation of MSM datasets in countries with fewer research resources; to facilitate dialogue between community, academic and public health sectors; and to maximise the educational impact on respondents of taking part in the survey—EMIS was a major opportunity to increase community knowledge about HIV.

Network Creation

The drive for a pan-European MSM survey came from a meeting in February 2007 organised by two of the authors (UM and AJS) working at the Robert Koch Institute (RKI, Berlin), which brought together researchers engaged in community surveying among MSM across Europe. Following this meeting, the RKI invited four organisations (in Germany, Italy, The Netherlands, Spain and UK) and a technical co-ordinator to be Associate Partners (APs) of EMIS.

The APs designed and sought funding for the survey from the Executive Agency for Health and Consumers (EAHC) of the European Commission. The first application having been declined, the APs redesigned and re-costed the project, and the second application was supported by the EAHC and was co-funded by the six APs. The Scientific Coordinator (AJS) took the lead in recruiting a further 77 collaborating partners (CPs) from community, academic and public health organisations across 35 European countries, aiming for one community and one either academic or public health partner in each country. In addition, three large and three smaller commercial gay dating websites were contracted with requests to support recruitment and dissemination and to advise on the acceptability of methods. Collectively, all partners are referred to as the EMIS Network.

EMIS was funded from March 2009 to September 2011. Funding facilitated the establishment of the EMIS Network, development of the questionnaire, a 3-month period of data collection, preparation and distribution of national and international data sets, and write-up in a variety of reports. Representatives of the EMIS Network met twice during the funding period, and the APs met a further four times. All other business was conducted electronically. The project was directed by a Steering Group of senior representatives from the six APs and supported by an Advisory Board consisting of two elected representatives from the CPs, and representatives of the ECDC and WHO/Europe. The project activity was divided into nine Work Packages each of which was designated a lead AP. An overall scientific co-ordinator was based at the Robert Koch Institute.

Sample Target

The population of concern to the EMIS Network is men living in Europe who have sex with men and/or feel attracted to men. This was operationalised in four inclusion criteria: self-identified as male; living in Europe; at or over the age of homosexual consent in country lived in; sexually attracted to men and/or had sex with a man in the last 12 months.

A description of the nature and purpose of the study was provided, but no assessment of whether participants understood it was undertaken. Participants were required to indicate that they understood these and that they consented to take part.

Questionnaire Design

The AP leading questionnaire development (Maastricht University) requested all EMIS partners to supply pre-existing national or regional questionnaires for MSM, in English. In total, 23 questionnaires from 20 countries were received. Individual questions were organised according to frequency, diversity, terms used, subjects and response formats. Literature reviews were conducted to ensure the scientific basis of the questionnaire was supported by psychological and behavioural theories and previous studies.

Previous questionnaires, core indicators, scientific literature, consultation with experts and feedback from the APs resulted in the first draft of a questionnaire presented to the first general meeting of the EMIS Network in December 2009. All topic areas and items in the first draft were considered by the Network using a floating roundtable system, with discussion focussing on acceptability (to collaborators) and prioritisation. In addition, the meeting reached agreements on four design areas: recall periods, informed consent, the lower bound age and range of languages.

Following the first general meeting, APs re-examined the draft and made extensive amendments. The content of draft 2 was organised in four key conceptual areas:

  1. 1.

    The levels and distributions of sexual HIV/STI exposure and transmission facilitators (‘behaviours’),

  2. 2.

    The levels and distributions of unmet sexual health needs (‘needs’),

  3. 3.

    The population coverage and acceptability of prevention interventions (‘intervention performance’),

  4. 4.

    The information needed to compare samples and target interventions (‘demographics’).

With a limited length and a large collaboration, many interesting questions were suggested that could not be asked. Pertinence to the above areas was a key criterion for consideration in the survey. The design sought a balance of questions across the four areas.

The funded proposal did not specify an intended completion time for the survey. Initial discussions centred on the desirability of 20 to 30 min as maximising data collection within a tolerable length based on previous experience. The first pretesting (see below) showed a wide variation in completion times with a central tendency toward the top of our desirable range. Although the variation diminished in subsequent testing and in the field, the median completion time dropped only slightly.

The survey required questions that were relevant for all MSM across differences in biological and social gender, sexual identity, and social and political living environments. Designing questions to collect demographic data was often challenging because they require different but comparable constructions in different countries. For example, information on migration history, ethnicity and religion were difficult to query across Europe, due to large differences in immigration history, immigration laws and minority concepts.

Draft 2 suggested an appropriate order for the questions. APs agreed to start and finish the survey with relatively neutral questions to reduce respondent discomfort and under-reporting.

Following broad agreement among APs, draft 2 was posted on the EMIS website in mid-January 2010. The Network was asked to provide feedback within 4 weeks on: the length; the balance of question topics; the acceptability of questions for specific countries and clarity of the (English) wording. Involving potential respondents in survey design can simplify data collection and analysis (Daley et al. 2003). CPs were asked to pilot the English questionnaire for comprehension and length by asking five MSM to complete it using paper-and-pencil and to record their feedback. Detailed comments were received from 21 EMIS partners. Completion times were obtained from 51 men and ranged from 10 to 49 min with a median of 30 min. In addition to highlighting numerous minor issues, this process identified survey length as a key area of concern. Modifications based on this feedback resulted in draft 3, a paper version of which was circulated to APs at the end of February 2010 and approval sought within 1 week. Only few (minor) changes were needed to reach AP approval of draft 3 for transfer online.

Online Transfer and Piloting

The benefits of self-administration to maximising valid responses to sensitive questions apply to online surveys: Specifically, respondents are less likely to over-report desirable behaviours and less likely to under-report socially undesirable behaviours because of the sense of anonymity and/or confidentiality afforded by online surveys (Bradburn et al. 2004).

The transfer of a paper survey to an online survey requires numerous small (and occasionally larger) modifications. The questionnaire was constructed within the chosen Internet survey software (www.demographix.com) in English. Eleven men in London were observed completing the survey online and responded to questions about how they answered the survey, to ensure correct interpretation of questions and to identify difficulties in completion. The resulting online version was sufficiently different from draft 3 to warrant being called draft 4.

Draft 4 online was shared with both APs and CPs at the end of March 2010, and all were invited to pretest it in English with MSM who had not yet seen the survey, focussing on: routing (serving or skipping questions based on previous answers); response sets that varied by country (region and education qualifications); the HIV-test setting response subset (since settings varied greatly); completion time and acceptability. Comments and completion times were collected from 76 online pilots with a median time of 26 min (range, 10–45 min). Comments on draft 4 were also received from 26 partners. In response, several changes were made online to create draft 5 which was discussed at the third Steering Group meeting in London (April 2010).

Survey length remained an area of disagreement between APs. Longer surveys can collect more data per case but have greater attrition, making findings less generalisable. This was the only design issue that required a vote by APs to resolve. The decision of the group was towards a longer survey with greater attrition. Some minor changes to draft 5 arising from Steering Group discussion were made, and all routing associated with the English-language questionnaire was checked by three researchers working independently. The final English language online version was signed-off by the APs at the end of April 2010. It sought 278 data items (although not all respondents were asked all questions), covered the six core ECDC indicators and nine of ten MSM-specific ECDC indicators (European Centre for Disease Prevention and Control. Mapping of HIV/STI Behavioural Surveillance in Europe. Stockholm 2009) and is available at http://emis-project.eu/research-questions-and-covered-items.

The survey was presented on 25 core pages and, depending on answers to some questions (e.g. country of birth, HIV testing history, sexual partners and substance use), another 18 sub-group pages. To minimise completion time, the survey was tailored using intra-questionnaire filters (routing) wherever possible. For example, questions regarding non-steady sexual partners were not shown if the respondent had already stated they had not had any non-steady partners.

Many EMIS partners were concerned about the sensitive nature of the data (including sexual behaviours and partial postal codes). This was expected to negatively affect overall response rates, item non-response rates and response accuracy (Tourangeau and Yan 2007), perhaps because men would be afraid that their data would be misused by third parties (Singer et al. 1995). To allay these fears, the opening page of the survey described the study aims and informed potential respondents that their data would be anonymous, that their privacy would be maintained in line with the European Data Protection Directive, that no IP-addresses or other data that could be used to identify computers was saved, and the survey software installed no cookies or any other trace files on computers. We told respondents this because anonymity in Internet-based surveys has been shown to reduce social desirability effects (Joinson 1999). As a consequence, however, respondents could not pause the questionnaire and sign-in later to finish it. This was considered an acceptable loss to ensure anonymity. Not collecting IPs also meant that it was possible for one person to submit two or more questionnaires (while the absence of material incentives and the length of the questionnaire act against this).

The age of consent being different in different countries posed a challenge for a unified approach. The solution was to ask respondents to declare that they understood the aims of the study and that they were old enough to legally have sex with men in the country they were living in. That men were at the legal age of consent was not tested for each country individually. Men were asked to check a box that said “In the country I live in, I am old enough to legally have sex with men” and were left to judge this for themselves. We wanted to ensure respondents were aware of the aims of the study and participated on a voluntary basis.

Finally, discussions about sexual vocabulary had reached no firm consensus across countries. Familiar wording is widely believed to increase reporting of socially ‘undesirable’ behaviour (Daley et al. 2003) and was common in the submitted national questionnaires used for draft 1. On the other hand, some collaborators felt the survey would not be taken seriously (have authority, or appear scientific) if it used slang vocabulary. It was decided to use both formal and informal terms wherever possible, with one following the other in parenthesis. The order of the terms and the colloquialisms used were determined by the country leads with experience in MSM surveying in their own countries. The English language version of the survey uses formal language first, followed by a colloquial paraphrase in parenthesis, for example: Did you ejaculate (cum) into his rectum (arse)?

Translation and Online Preparation

Translations were carried out online, using the survey hosting software to display the English version on the left half of a screen and a duplicate on the right half to be over-written with the translation. This process minimised routing errors and copy-and-paste errors.

The survey was translated into 24 further languages, including 20 of the 23 official languages of the EU. It was not available in Maltese and Slovak (no EMIS partner had been established in either of these countries at the time of translation) or Gaelic Irish (as it is a geographically concentrated minority first language in the Republic of Ireland). In addition to these 20 official EU languages, EMIS was available in Norwegian and Ukrainian (additional funding was available for these non-EU languages), Russian (a minority language in Poland, Lithuania, Latvia and Estonia, and the most frequent immigrant language in the EU), Turkish (a minority language in Bulgaria and second most frequent immigrant language in the EU) and Serbian (a minority language in Hungary and intelligible to many EU immigrants from the former Yugoslav states). Regrettably, the partner from the Former Yugoslavian Republic of Macedonia withdrew from the EMIS Network when we were unable to identify resources for a Macedonian language version.

Translation was an interactive process involving native-speaking stakeholders from the EMIS Network and two native-speaker translators for each language. We also involved several multi-language proof-readers to compare the translations with the English original and with each other. The proof-readers ensured a harmonised multi-language questionnaire while deliberately maintaining certain differences, identified as culturally appropriate, such as explicitness of language or addressing respondents formally or informally. In all languages, simple rather than specialised terms were aimed for (Dillman 2000).

APs checked the translated versions for visual integrity and layout online and ensured that all language versions were structurally identical and had the same routing between questions, and all saved their data in an identical format. CPs confirmed the terminology fitted with their perceptions of the norm for the target group in their country, reviewed the final survey and signed-off their main language version by the end of May 2010.

Response Monitoring

A 3-month field work period in June–August 2010 was chosen early in the planning process other planning milestones slotted into place before it. The length and specific months were chosen to fit with the largest number of pre-existing behavioural surveillance tools in operation across Europe.

EMIS was available for completion online for 12 weeks, between 17:00 h (British Summer Time) on Friday 4th June and midnight on Tuesday 31st August 2010. The survey was functionally identical in each of the 25 language versions, but each version had its own Uniform Resource Locator (URL, the global address of documents and other resources on the World Wide Web). While each URL could be accessed directly, all advertisements and invites directed users to a dedicated landing page, hosted alongside the survey software. This landing page presented the 25 language names (‘English’, ‘Deutsch’, ‘Čeština’, etc.) in a five-by-five rectangle and carried a counter for the total number of surveys submitted to date across the 25 languages. By clicking directly on a language name, respondents were taken to the respective version of the questionnaire.

Our online survey provider stored all incoming data on multiple secure and encrypted data servers. These were backed-up daily and hourly at times of the highest volume of incoming data. Page view data were captured by our survey software, allowing estimates of attrition across the survey. Respondent-derived data (answers) were only transferred to our servers when the respondent clicked ‘submit answers’ at the end. Data from respondents who broke-off or who did not submit at the end of the survey were not captured. This would have been feasible (if technically complex) by using cookies to store data as it was input. However, doing so would compromise our ensuring the anonymity of respondents and was decided against.

Following submission, all respondents were sent to an HIV prevention website appropriate to the language of survey completion and country of residence. Exit websites were selected by EMIS partners.

Our survey software provider produced a consolidated version of the 25 surveys so responses could be monitored in real-time, both within any of the 25 language versions and in a single consolidated database. National lead partners had real-time online access to data collected that enabled them to see how many respondents had accrued from each promotional activity and conduct some descriptive analyses as fieldwork progressed. The AP with overall responsibility for survey promotion also monitored national response rates daily during the first 6 weeks and then weekly to the end of the survey. This centralised and decentralised monitoring of responses was used to plan specific promotions to stimulate recruitment in countries where lower responses than projected were observed.

Promotion

Planning for the promotion of EMIS to potential respondents began 6 months prior to its launch. APs agreed that promotion would target national and transnational commercial and NGO websites, social networking websites and blogs, with printed posters for display in community venues, and business cards for hand-to-hand distribution. The APs committed to providing promotional materials in all 25 survey languages if requested.

During the latter stages of questionnaire development, all partners were asked to identify appropriate national and supranational websites for promoting EMIS. Representatives from one of the APs (Sigma) supported national CPs to establish contact with webmasters and reach agreements about the type of advertisement and promotion that was feasible, and fees payable, aiming to guarantee maximum visibility for minimum investment. If a fee was requested, the promotion lead AP (Sigma) liaised directly with the webmasters to ensure payment was made, and a precise contract was written and signed.

All websites promoting EMIS were allocated a specific URL for use on all their online advertisements. This URL took people to the EMIS landing page, and when they made their language selection, the source URL was captured as the first item of data from that respondent. Consequently, we know through which websites respondents were recruited. Overall, at least 237 unique websites recruited to EMIS of which 22 (9 %) required payment.

As a final method of recruitment, on submission, respondents were asked to nominate up to three friends to invite via E-mail to complete the survey.

Visual Identity, Buttons and Banners

After consultation and debate at the first general meeting, the Steering Group agreed on the broad principles for the visual identity of EMIS. Notions of community, inclusion and participation were central to these requirements. Thereafter, all partners were included in two online polls to establish preferences for a visual identity. The agreed visual identity was then used to develop the printed (offline) promotional materials and the online buttons and banners for websites.

All partners were polled for the choice of a core promotional slogan. The English version of the winning slogan was “Be part of something huge!” and an image of people putting their hands up, conveying the large-scale communitarian nature of the survey. In English rotating banners, the slogan alternated with the words “EMIS Men’s Sex Survey” and “Do me now!” in order to be both intriguing and mildly sexually suggestive. Some National EMIS partners chose to modify the slogan to best match vernacular characteristics.

A large number of clickable promotional banners and buttons were developed for use on websites. The promotion lead AP asked partners and prospective advertisers to confirm the number, size and specification of buttons and banners required and co-ordinated their production by Sparkloop®. Ultimately, 191 different buttons and banners were produced in 24 languages. These are available at www.sparkloop.com/visuals/emis. Clickable banner advertisements were used on national and transnational websites, some through paid advertising, but the majority carried at no charge.

Trans-national Online Promotion

In most countries, the largest proportion of respondents were recruited from five pan-European gay commercial and community websites sending instant messages (IMs) to their members and another four websites placing prominent banner advertisements for EMIS. Among the eight trans-national websites paid to promote EMIS charges varied considerably, with no obvious relationship between agreed costs and the number of recruits arising from any specific advertiser.

IMs consist of a short text and/or picture to the users’ personal message box describing and endorsing the survey. Five international websites were paid to send IMs: PlanetRomeo®, Manhunt/Manhunt Cares® and Gaydar®, each of which has membership across Europe; Qguys® for countries within the Commonwealth of Independent States and the Baltic countries and Qruiser® for Scandinavia. The invitation to take part was drafted by the APs, modified by the website representatives and agreed by both parties. The invitations came from and were signed by the websites. A typical invitation opened with the words “Because we know you are committed to getting the best sex with the least harm, we want to introduce you to questionnaire about sex, health and relationships”. The invitation then stressed the size and safety of the survey, suggested it would take about 25 min before offering potential respondents an exchange:

“Taking part might mean you find out something new and the information you provide will help programmes better meet our needs. A high response rate will also help us send a signal to governments that HIV and sexual health are still very important to the gay community. It could also mean that services for gay and bisexual men get funded.”

The invitation was crafted to maximise impressions of inclusion and impact by associating participation with positive and feasible community outcomes.

PlanetRomeo® was the single largest EMIS recruiter, advocating for the survey through multiple channels and recruiting 103,000 men. It was the first website to be contracted to send IMs, which were temporally staggered to 1,060,772 members across all target countries. IMs were sent in 25 different languages, matched to the member’s profile. Response rates to the PlanetRomeo® IMs varied by country from 5 % to 15 %, with a median response rate of 10 %. Later, during the fieldwork, the site carried a homepage article about the survey and a survey advertisement banner. PlanetRomeo® recruited more than 75 % of the sample in ten countries and 50–75 % in a further eight countries (see Table 1).

Table 1 Recruitment websites, proportion recruited through major websites and small media used for advertisements, by country

Manhunt/Manhunt Cares® recruited over 12,000 respondents from the second tranche of IMs. These were sent in six languages (English, French, German, Italian, Spanish and Portuguese) to 181,000 members during weeks 4 and 5 of data collection. A sub-tranche of IMs was sent in the last week of recruitment to men who had joined Manhunt since the first messages were sent out. Manhunt/Manhunt Cares® accounted for over 50 % of respondents in Portugal and substantial proportions in Spain, Republic of Ireland and the UK (see Table 1).

Gaydar® recruited 11,000 men, having undertaken the third large-scale international paid advertising, sending IMs and carrying banner advertisements during the last 5 weeks of the recruitment period. Gaydar® was asked to target those countries which had not yet reached three respondents per 10,000 general population. Five weeks before the end of fieldwork, these were Bulgaria, Czech Republic, France, Hungary, Ireland, Latvia, Lithuania, Moldova, The Netherlands, Poland, Portugal, Romania, Russia, Serbia, Sweden, Turkey, UK and Ukraine. This flexible strategy had variable success. Gaydar® accounted for almost half of the UK respondents and significant proportions in Ireland and Portugal (see Table 1).

In the latter half of the recruitment period, Qguys® was contracted to send IMs to all its members in the Russian Federation, Belarus, Estonia, Latvia, Lithuania, Moldova and Ukraine. IMs were sent in either Russian and English or Latvian and English, depending on the language of registration of members. Almost two thirds (63 %) of the 2,800 men recruited by Qguys® lived in the Russian Federation, a quarter (27 %) in Ukraine and a small proportion (5 %) in Belarus.

Similarly, Qruiser® was contracted to place banner advertising on its website and later to deliver pop-up messages to its members in Sweden, Finland, Norway and Denmark, matching the language in the message with the preferred language of the member. The vast majority (90 %) of the 2,377 men recruited through Qruiser® lived in Sweden with the bulk of the remainder in Finland.

Three other international websites recruited to EMIS using banner advertisements only, and none recruited more than 1,000 men. Banner advertising was purchased on Recon® and Barebackcity® in response to requests from a number of EMIS partners; however, the majority of respondents from these two sites lived in the UK and Germany, respectively. Finally, the main website of the International Lesbian, Gay, Bisexual and Intersex Association provided free advertising and recruited more than 100 men resident in a range of European countries.

During the last month, a final method of targeted recruitment using Google Adwords®, presented targeted (paid) advertisements to people who used specific phrases in the Google search engine. Advertising was targeted at men in countries where respondents per 10,000 total population was still below 1 (Poland, Romania, Russia, Slovakia and Turkey) but with limited success, recruiting 315 completed qualifying surveys across these five countries.

In addition to advertising, promotion was attempted in most participating countries using Facebook®, the world’s largest online social network site. Facebook® promotion included the establishment of EMIS Event Pages in all countries and targeted approaches to popular opinion leaders, HIV and LGB organisations and gay commercial organisations. All these people and organisations were asked to promote EMIS though their Facebook® networks. This free but very time-intensive approach did not result in a large number of respondents overall (approximately 1,500) but contributed a reasonable number of men in some countries, especially Sweden and Italy, but also Denmark, UK, Slovakia and Belgium.

National Online and Offline Promotion

At least another 227 national websites carried buttons or banners and recruited to the survey (some agencies did not seek a unique URL for their site but instead copied one from another website, so the actual number of sites recruiting to the survey will be above this number). Most national HIV and LGBT NGOs supported the project, usually at no cost. Fifteen national websites were paid for their promotion, including sites in Austria, Bulgaria, Denmark, Greece, Ireland, Italy, Latvia, Lithuania, The Netherlands, Romania, Slovenia, Spain and Switzerland. In some countries (e.g. Germany, Sweden, UK) partners financed additional national advertising. While paid-for advertisements were usually more productive than free advertisements, there was no direct relationship between the cost of advertising and the volume of banners served, or the number of men recruited. All requests from Collaborating Partners for funds for national advertising were fulfilled. National websites recruited more than 50 % of respondents in 13 countries: Bulgaria, Belarus, Denmark, Czech Republic, Estonia, Finland, Lithuania, Latvia, Moldova, Norway, Russia, Slovakia and Slovenia. Some national websites recruited men beyond their own borders, for example, a Czech website recruiting men living in Slovakia.

Partners complemented online promotion by distributing paper materials in a range of gay community settings. Materials, including business cards and posters, were developed by a lead AP in collaboration with CPs, who specified dimensions and quantities. The AP and CPs formulated precise wordings for cards and posters. Sparkloop® was commissioned to produce the materials. Many partners included referral information to national services on their promotional materials. Overall, 27 different versions of the business card were produced, and 139,350 were printed and delivered to 37 EMIS partners. In addition, 23 versions of the poster were produced, and 6,635 were printed and delivered to 24 EMIS partners. EMIS partners then often shared them with other agencies in their countries increasing dispersal of offline survey promotion.

No country-specific recruitment was undertaken in five countries (Cyprus, Malta, Bosnia, Luxembourg and Croatia) as there was no active CP there at the time of data collection. CPs in a further four countries (Greece, Hungary, Serbia and Slovakia) identified no suitable country-specific websites. In Turkey, only local and regional, but no national, LGBT organisations exist. Due to the terms of the grant, non-EU countries were ineligible to draw on the EMIS budget for promotion costs. Thus, for many of these 11 countries, response rates (based on total population size) were lower than elsewhere.

Attrition Across the Language Versions

Figure 1 shows the proportion of men who continued with the survey at each of the 25 core pages. The first page of the questionnaire constituted the introduction to EMIS and represents 100 % in the graph. The proportion of respondents served each of the subsequent 24 core pages is plotted on the graph. The proportion of respondents who proceeded from page 1 to page 2 (by confirming that they had read the introductory text, consented to participate and were old enough to legally have sex with men in their country of residence), varied from 36 % (Slovenian) to 76 % (English) and accounts for the majority of the total attrition across the survey. Slovenian stood out from all other languages in the size of this drop, most likely because the most productive Slovenian promotional site was not gay-specific but a generic dating website with MSM sections, therefore many people who accessed the introductory page were not MSM.

Fig. 1
figure 1

Attrition across survey pages by the 25 languages

Almost all respondents in all languages moved from page 2 (the first page of questions) to page 3. Of those presented with page 2, the proportion who reached the 25th page (‘Submit’) ranged from 62 % (in Turkish) to 76 % (in Norwegian), with a mean of 68.5 % across the 25 languages.

Total Returns

There were 184,469 cases in the consolidated EMIS data file at the close of data collection. When downloaded, three cases were found to have been created by incorrect kerning in the survey software (the creation of another case by misreading of a comma in an open-ended response), resulting in 184,466 cases. Of these, 464 cases stated no country of residence, and another 1,963 stated a country of residence outside of Europe and were removed from the datasets leaving 182,039 cases (98.7 % of submitted) known to be living in Europe.

Non-qualifiers

Non-qualifiers are respondents who did not meet the criteria for inclusion in the study. The number of cases submitted and the number of non-qualifiers in each country are presented in Table 2. Non-qualifying cases included:

  • Two cases did not check that they read and understood the introduction;

  • 279 cases indicated they were women (but not transgender women);

  • 196 cases provided no evidence for homosexual desire, gay/bisexual identity or sexual behaviour with men

  • 303 cases gave no numeric value for age;

  • 24 cases gave an age between 1 and 12;

  • 33 cases gave an age over 89.

Table 2 Cases submitted and non-qualifiers by country of residence

Some cases were disqualified on more than one criterion (for example, being both a woman and missing age); therefore, the total number of non-qualifying cases is less than the sum of the exclusions. A total of 544 cases living in Europe were excluded, leaving 181,495 cases who met the qualifying criteria.

Datasets

EMIS data can be divided into national datasets based on current country of residence, regardless of the language used to complete the survey or country of birth, and combined into a pan-European dataset. National datasets have been made available for all 38 countries with 100 or more qualifying cases (those countries’ names not in brackets in Table 2). Thirteen European countries and states did not reach 100 qualifying cases: Albania; Andorra; Armenia; Azerbaijan; Georgia; Iceland; Kazakhstan; Kosovo; Liechtenstein; Monaco; Montenegro; San Marino and Vatican City. Data from the 291 MSM living in these countries or states are not included in any dataset.

The pan-European dataset includes respondents from all of the 38 countries in Europe with 100 or more qualifying cases.

Discrepant Data Flags

In several places, the questionnaire allowed logically inconsistent data to be supplied, where answers to two questions cannot both be valid. Inconsistent data could be submitted by moving backwards and forwards in the survey and changing previously given answers and simply by supplying inconsistent answers in different parts of the survey. To increase the quality of the data, we constructed six ‘discrepancy flags’ which indicated whether the respondent had supplied inconsistent data in six areas: age (six possible inconsistencies), HIV testing history (four possible inconsistencies), STI testing (three possible inconsistencies), sexual practices (seven possible inconsistencies), steady partners (seven possible inconsistencies) and non-steady partners (13 possible inconsistencies). Overall, 14.2 % of qualifying cases had one or more discrepancies (the maximum number observed was 11 out of a possible 30). National databases contained all cases so national leads can make exclusions according to their own needs. To strike a balance with case retention, the APs agreed that pan-European data analyses will exclude cases with two or more discrepancies, excluding 3.7 % of qualifying cases in EMIS countries. Hence, if the sample includes only residents of the 38 countries with more than 100 eligible respondents, and excludes all men where two or more data discrepancies were observed, data from 174,209 men are eligible for analysis.

Discussion

EMIS collected comparable data in 25 languages, advertised and promoted on at least 237 national and trans-national websites for MSM. It was the first pan-European MSM survey using a multilingual questionnaire and comparable recruitment procedures across a large number of countries. EMIS results are being jointly analysed with a view to a common understanding of HIV prevention challenges and to foster cooperation between sectors and agencies. These benefits are particularly valuable for new EU member states in which MSM communities and HIV prevention responses are less well established and where policy supporting MSM HIV prevention might be less developed.

The drivers for the survey were the researchers themselves rather than the funder or other pan-European co-ordinating bodies, and it was the APs who managed and negotiated the competing demands of the wide-ranging stakeholders. While some additional national government finance was provided to extend the survey to non-EU countries, little other support or obstacles were encountered from governments. We have demonstrated the feasibility of such a project and the validity of its HIV prevalence data (Marcus et al. 2012). Future policy makers can foster similar future international collaborations by recognising their value and ensuring international collaborations have as wide a reach as possible. The cost savings on generating data collectively in this way is considerable.

Estimating the time the project took is difficult as no one clear inception point can be identified. One possible seed was a meeting in September 2005 meeting organised by one of the current authors (HH) on joint MSM monitoring studies, with researchers from four countries. This seed was fostered by two further authors (UM and AJS) following a February 2007 meeting of several countries organised by them and driven through two funding applications. The project was funded from March 2009 to September 2011. The first journal article was published at the end of 2012. So from initial ideas to publication took 7 years, within which 2½ were funded. We note that publication is part of the research process, not its end point and that EMIS is not over.

There are no reasonably sized known representative samples of MSM to which EMIS respondents could be compared. However, these national samples may be more representative of (intra-European) migrant MSM than any previous national surveys because all men across Europe could complete the survey in any of 25 languages.

The two principal sources of bias in the data are uneven access to the Internet across and within countries and the self-selection bias in the recruitment process (Evans et al. 2007). Household access to the Internet varies considerably across Europe and is generally far more common in the western and northern parts of Europe than in eastern and southern parts (International Telecom 2011). Figure 2 shows the proportion of all households with internet access in 2009 by country (countries in brackets had no EMIS partner at the time of recruitment). Among the 38 countries from where sufficient respondents were recruited to warrant analysis, less than 20 % of households had Internet access in four countries (Moldova, Ukraine, Bosnia and Herzegovina and Belarus), and less than 33 % of households had access in another seven countries (Bulgaria, Turkey, Serbia, Macedonia, Russia, Romania, Greece). Consequently, the samples from these countries were smaller and may be less representative than in other countries.

Fig. 2
figure 2

Household internet access in 2009, by country (Source: (International Telecom 2011).) Countries in parenthesis are those for which EMIS had no national partner at the time of recruitment

Although PlanetRomeo® contributed hugely to recruitment to EMIS, the project was not dependent on this site. Excluding all men recruited through PlanetRomeo®, we would still have recruited 75,909 qualifying cases, with a minimum of 100 qualifiers (our target for individual countries) in 30 countries and over 1,000 qualifiers in 16 of those countries.

Usability testing was limited to the English language version. Draft 2 was pretested in English on paper with men in several countries and draft 4 pretested in English online with several men in London. Translations concentrated on comparable meanings but other online language versions were not pretested. We acknowledge that the exporting of measurement tools from one cultural and linguistic group to another is a process that is vulnerable to cultural differences and translation problems. To ensure that translated items are equivalent to their original versions, both statistical and qualitative analyses are necessary (Ware et al. 1995; Auchter & Stansfield 1997). We did not rigorously test the survey in other languages, and this may have introduced differential data validity across language versions, with the English version generating more valid data.

Overall, 68.5 % of respondents who viewed the first page of questions made it to the end of the survey and submitted their answers, and 31.5 % of respondents aborted the survey before the end. This proportion is directly related to the length of the survey and the number of items. It remains a matter of debate about the relative merits of more information from a smaller and less representative number of men or less information from a larger and more representative group. Given our lack of knowledge about how far from representation these samples are, it is difficult to assess the importance of this loss of representation. However, from the perspective of user satisfaction, submitting answers will usually be better than abandoning the survey. While several external reasons may arise for abandoning the survey, internal reasons will rarely be because it is too good. They more likely to be because it is boring, too long, perceived to be irrelevant, offensive or insensitive. While respondents reaching the end of a survey and submitting their answers does not guarantee their satisfaction, the proportion of users who abandon the survey can be an indication of dissatisfaction. In addition, since ‘making a difference’ is a key motivation for taking part in social research, abandoned surveys may be dissatisfying because they clearly cannot go towards making any difference as their data were not captured.

This would suggest weighting the value of shorter surveys. However, it is worth noting that, in Fig. 2, attrition is steeper in the first half of the survey compared with the second half. This suggests most attrition occurs in the first few questions and declines over the course of the survey. At the length it was, the survey would have had to be very considerably shorter in order to retain a significantly larger proportion of respondents. Following completion of page 2 of the survey, attrition across the language version was within a relatively narrow band (62–76 %). This suggests little difference in the acceptability of the different language versions, including their varying use of colloquial and formal vocabulary for sex and body parts.

In order to facilitate research collaborations, commissioners of community-based surveying should consider the purpose of such surveys and specify their intended lengths (or intended completion times). Clear and detailed specification of the data collection objectives for individual items as well as the entire survey can contain survey length and identify admittedly interesting (even valuable) questions that really belong elsewhere.

The coverage and depth of recruitment achieved across Europe were the result of meaningful involvement of a large number of collaborators in the survey design, its visual identity and in promotional strategies. Policy makers cannot dictate successful community and inter-agency collaborations in research. Commissioners of surveys can however foster such positive outcomes through encouraging projects to include a range of stakeholders, to make explicit their research values and objectives and to define transparent development and decision making processes. In our international collaboration, flexibility in accommodating a range of differing national needs was essential to maintaining the network and to the appropriateness of the survey across a range of settings.

While the overall sample size and distribution is one milestone towards success, we also consider the supply of national data sets to 38 country-level partnerships to be an indicator of the success of the project. Thirdly, the probable validity of EMIS HIV prevalence measures has now been demonstrated (Marcus et al. 2012): EMIS therefore provides HIV prevalence estimates among MSM for several countries that did not previously have such estimates. And fourthly, given the similarity of methods in each country, EMIS provides an unprecedented cross-Europe picture of MSM and HIV for planners and policy makers that is the primary outcome for the study.

Careful design, piloting and presentation ensured the survey was acceptable and had both authority and perceived community benefit. To reach substantial coverage across Europe, a patchwork of recruitment was required across a large number of commercial and community partners. A high degree of partner involvement in the recruitment efforts and commitment of the commercial partners were crucial to the feasibility of Europe-wide surveying. The large number of participating websites (along with the small value of payments agreed by local partners) contributed to this feasibility and constitutes an important strategy to consider in research areas with limited public funding. Recognising and accommodating the range of uses a diverse group of stakeholders find for community-based surveying was central to maintaining this wide-ranging collaboration.