Article Text

Download PDFPDF

Use of the nominal group technique to identify stakeholder priorities and inform survey development: an example with informal caregivers of people with scleroderma
  1. Danielle B Rice1,2,
  2. Mara Cañedo-Ayala1,
  3. Kimberly A Turner1,
  4. Stephanie T Gumuchian3,
  5. Vanessa L Malcarne4,5,
  6. Mariët Hagedoorn6,
  7. Brett D Thombs1,2,3,7,8,9
  8. Scleroderma Caregiver Advisory Team
    1. 1 Lady Davis Institute for Medical Research, Jewish General Hospital, Montreal, Quebec, Canada
    2. 2 Department of Psychology, McGill University, Montreal, Quebec, Canada
    3. 3 Department of Educational and Counselling Psychology, McGill University, Montreal, Quebec, Canada
    4. 4 Department of Psychology, San Diego State University, San Diego, California, USA
    5. 5 San Diego Joint Doctoral Program in Clinical Psychology, San Diego State University and the University of California, San Diego, California, USA
    6. 6 University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
    7. 7 Department of Psychiatry, McGill University, Montreal, Quebec, Canada
    8. 8 Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, Quebec, Canada
    9. 9 Department of Medicine, McGill University, Montreal, Quebec, Canada
    1. Correspondence to Dr Brett D Thombs; brett.thombs{at}


    Objectives The nominal group technique (NGT) allows stakeholders to directly generate items for needs assessment surveys. The objective was to demonstrate the use of NGT discussions to develop survey items on (1) challenges experienced by informal caregivers of people living with systemic sclerosis (SSc) and (2) preferences for support services.

    Design Three NGT groups were conducted. In each group, participants generated lists of challenges and preferred formats for support services. Participants shared items, and a master list was compiled, then reviewed by participants to remove or merge overlapping items. Once a final list of items was generated, participants independently rated challenges on a scale from 1 (not at all important) to 10 (extremely important) and support services on a scale from 1 (not at all likely to use) to 10 (very likely to use). Lists generated in the NGT discussions were subsequently reviewed and integrated into a single list by research team members.

    Setting SSc patient conferences held in the USA and Canada.

    Participants Informal caregivers who previously or currently were providing care for a family member or friend with SSc.

    Results A total of six men and seven women participated in the NGT discussions. Mean age was 59.8 years (SD=12.6). Participants provided care for a partner (n=8), parent (n=1), child (n=2) or friend (n=2). A list of 61 unique challenges was generated with challenges related to gaps in information, resources and support needs identified most frequently. A list of 18 unique support services was generated; most involved online or in-person delivery of emotional support and educational material about SSc.

    Conclusions The NGT was an efficient method for obtaining survey items directly from SSc caregivers on important challenges and preferences for support services.

    • systemic sclerosis
    • scleroderma
    • nominal group technique
    • caregiver

    This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

    View Full Text

    Statistics from

    Strengths and limitations of this study

    • This is the first study to use the nominal group technique to investigate the experiences of caregivers of people living with scleroderma.

    • Participating caregivers were asked to deliver answers in a round-robin format, which gave all participants an equal opportunity to describe the challenges they face and their support service needs and preferences.

    • A Scleroderma Caregiver Advisory Team, comprising caregivers of people living with scleroderma, was involved at each stage of the study.

    • Participants were recruited from scleroderma national patient conferences, which may limit the generalisability of findings.

    • Findings will need to be verified with a larger number of caregivers in survey format due to the relatively small number of participants.


    Surveys provide a feasible method for gathering and prioritising input from large numbers of stakeholders in order to inform programme development.1 A number of approaches can be used to develop survey items for the purpose of needs prioritisation. Examples include adapting a pre-existing questionnaire originally designed for use with other groups, gathering expert opinions and using theoretical definitions of a construct to guide item generation.2–5 These approaches do not explicitly integrate perspectives of stakeholders, however, and risk identification of programme goals that may not be ideally aligned with the needs of the target group.6 Qualitative focus groups and individual patient interviews may also be used to generate items. These methods, however, can be time-intensive and resource-intensive and may overly emphasise input from some participants and not others.7 Furthermore, they require researchers to translate identified themes into survey items, but do not directly generate survey items.

    The nominal group technique (NGT) is an approach that can be used to structure group discussions in a way that allows stakeholders to directly generate items for a needs assessment survey. The NGT method facilitates generation of survey items in a straightforward, cost-efficient manner by directly soliciting stakeholder input on survey items to address specific research questions.8 9 In the context of needs assessment, it can be used to create a priority list of challenges that need to be addressed, along with potential solutions. When the NGT is used, a specific question is presented to the group of participants, then participants individually and silently generate lists of examples in response to the question. Following this, each participant shares each of her or his items one at a time in a round-robin format until all items are shared, which results in a compiled list of items from the entire group. After the comprehensive list is compiled, group discussion occurs among participants and items are removed, reworded or added to the list. Finally, participants vote or rank the items generated in terms of importance or relevance.8 The NGT has been used previously for preliminary item generation and to provide direction for survey development with stakeholder groups that include patients and family members of persons impacted by health conditions,10 including people with type 2 diabetes and multiple sclerosis, and caregivers of people with Parkinson’s disease.8 10–12 In addition to its efficiency and ability to directly incorporate stakeholder input into surveys, the collaborative nature of the NGT may increase stakeholder ownership of research and increase the likelihood that programmes that are developed effectively address stakeholders’ most important needs.13

    Informal caregivers are people who provide support for a family member or friend in need of care due to a health condition.14 There is only limited research on informal caregivers in rare diseases, despite the important role they fill and the emotional and practical challenges they face.15 Many patients with rare diseases have substantial care needs, but there are often few specialised resources available to them through the healthcare system. As such, informal caregivers for patients with rare diseases may undertake a substantial role in caregiving.16–18

    Systemic sclerosis (SSc, scleroderma) is a rare chronic autoimmune disorder, characterised by vascular damage, inflammatory system activation and excessive production of collagen.19 Only one doctoral thesis has considered the experiences of informal caregivers of people with SSc. This unpublished thesis was a qualitative study that included 13 caregivers.20 Consistent with caregiving in other diseases, SSc caregivers who participated in the study reported that their experience as a caregiver involved having to manage additional tasks (eg, household chores), increased negative feelings (eg, guilt) and personal stress, and relationship changes (eg, relational strain with the person with SSc).19 The specific challenges faced by caregivers of persons with SSc, however, have not been studied systematically.

    Developing resources for informal caregivers for persons with SSc may help them manage their role and reduce burden, but information is required on the challenges they face and their preferred support resources. Gathering information from a large number of caregivers is best accomplished via a survey. The objective of this study was to use the NGT in a series of discussions to develop survey items to assess: (1) challenges experienced by informal caregivers of people living with SSc and (2) their preferences for types of support services that could potentially be developed.


    Participants and procedures

    We conducted three NGT discussions at the national patient conferences of the Scleroderma Foundation of the USA (two groups) and Scleroderma Canada (one group). Eligible participants were people who had provided unpaid care in the past 12 months to a friend, family member or partner with SSc. It was not required that the caregiver live with the person diagnosed with SSc. Potential participants were emailed study announcements by the Scleroderma Foundation and Scleroderma Canada prior to the conferences. Participants who expressed interest in the study were then contacted by email by the study coordinator who provided them with details about the study, including information about the date and time of NGT discussions, researchers’ credentials, study goals, research questions and the end goal for the project. Prior to each group, paper copies of the consent form were provided to participants, and they were given the opportunity to ask questions about the study.

    Prior to beginning the NGT discussions, participants completed a brief demographic questionnaire that was used to describe participant characteristics. The questionnaire included items about the participants’ age, sex, race/ethnicity, employment status and information about the person for whom they provided care, including SSc diagnosis subtype and years since diagnosis. They also provided caregiving information, including type and length of relationship to the person with SSc, tasks undertaken as part of caregiving and time spent providing care.

    The three NGT discussions ranged in length from 90 to 120 min and were conducted in July 2016 (Scleroderma Foundation) and September 2016 (Scleroderma Canada). The three groups were held in private hotel conference rooms and were moderated by two members of the research team. The first and second groups were moderated by the principal investigator, a male professor trained in clinical psychology and knowledgeable about SSc (BDT) and a female doctoral student in clinical psychology knowledgeable about SSc (DBR). The third group was moderated by DBR and a female doctoral student in counselling psychology with experience in SSc research (STG). A female research assistant was present as an observer in the third group. All moderators had previous experience with discussion-based research. In addition, moderators pilot tested the interview guide among members of a research group from Montreal, Quebec, prior to conducting the caregiver NGT discussions. The pilot test did not result in changes to the protocol, but it resulted in increasing the time allotted for the planned NGT discussions. Participants were informed that the objectives of the NGT discussions were to (1) develop a list of challenges they faced as informal caregivers and (2) develop a list of caregiver support resources that would be helpful to them, as well as the ideal format for delivering these services.

    After the NGT procedures were explained, participants were presented with the first research question: ‘Think about the challenges you have faced since taking on a caregiving role to somebody close to you with SSc’. After being presented with this question, participants were asked to list on a piece of paper the challenges that they have experienced while helping to care for a family member, friend or partner with SSc. Participants developed a list of challenges individually without consultation with other group members. Once completed, participants shared one item at a time from their list in a round-robin format. Each member shared an item in turn until all members had a chance to share an item, then the process began again until every item on each participant’s list had been shared. Participants were instructed not to repeat items that were verbatim to items provided by others but to share any items that seemed to differ, even if only minimally. Participants’ answers were typed on a computer and projected onto a screen as they were provided so that the list could be viewed by the moderators and participants as it developed. Once all items had been shared, group discussion was used to clarify the meaning and wording of items and to remove or merge overlapping items. A consensus process was used, which involved asking participants if they agreed with the wording of items and, for example, if participants thought an item would be better separated into two items. Items were edited based on group feedback until agreement was reached for all decisions.

    In each group, once a list of unique items was agreed on, one of the moderators printed the list of items. Participants then rated the importance of each challenge listed on a scale from 1 to 10, with 1 representing challenges that they did not perceive as personally important to them in their role as an informal caregiver and 10 representing extremely important challenges. After each participant had rated the items, the moderator collected the ratings. Item ratings were collected to inform the removal of items that may have been suggested, but were not especially relevant to caregivers.

    Next, participants were presented with the second research question: ‘Think about services that could be put in place to provide better support to SSc caregivers. What programmes, services or supports would be helpful in your role as a caregiver? How would these programmes, services or other supports operate?’ Participants were asked to write down on a piece of paper any support services that they thought would be helpful and the way in which these services could be delivered. The same process used for answering the first research question was then applied to this research question, and a final master list of support services was developed. As with the first research question, each idea for a support service that was generated was rated independently by each participant on a scale from 1 to 10, with 1 representing support services that they believed they would be unlikely to use and 10 representing services they believed they would very likely use.

    For both research questions, as participants shared their items, if clarification was necessary, probes were used to gain a clearer understanding of the challenges and support services stated (eg, ‘Can you elaborate on that?’ (see online supplementary appendix S1 for interview guide)).

    Supplementary file 1

    Data analysis

    Descriptive analyses were conducted to present demographic data. Many of the challenges and support services items were generated in more than one group. Thus, the research team created a master list that combined all generated items, identified items that overlapped between groups and merged overlapping items. Mean scores were calculated for each unique item. All analyses were conducted with SPSS V.22.0.

    The master list of potential survey items that integrated responses from all three NGT discussions was distributed to members of the research team, including the Scleroderma Caregiver Advisory Team, which comprised eight  informal caregivers of people with SSc. All research team members reviewed the list of items and associated means. The team then made recommendations to remove items that were too vague or repetitive and had the opportunity to suggest new items. An iterative process was used to reword or remove any unclear items, and to incorporate any additional items deemed relevant and important, until consensus on a final list of items was reached.

    To categorise challenges, qualitative content analysis was employed.21 First, relevant literature was reviewed to consider pre-established categories for caregiver challenges. A previous systematic review of 192 articles categorised consequences of caring for people living with cancer into four categories.22 The four categories were used as a starting point to generate categorisations for the challenge items in our study. Challenges were first categorised by two members of the research team who, in collaboration with the principal investigator, developed definitions and rules for each category and refined or added categories as necessary (see online supplementary appendix S2). Another member of the research team, blind to the initial categorisation process, then used the definitions and rules to independently categorise the challenges.


    Participant and caregiving characteristics

    A total of 13 informal caregivers of persons with SSc (seven women, six men) participated in one of the three NGT discussions (Scleroderma Foundation Group 1=two women; 0 men; Scleroderma Foundation Group 2=three women; two men; Scleroderma Canada Group=two women, four men). None of the participants who came to the group sessions declined to participate or dropped out prior to completing the study.

    Participant sociodemographic and caregiving situation characteristics are presented in table 1. Caregivers ranged in age from 28 to 76 years (mean=59.8 years, SD=12.6). Most caregivers were employed full-time (n=5) or retired (n=7); one caregiver was unemployed. All five caregivers who worked reported that providing care interfered with their job. Participants provided caregiving for a partner (n=8), parent (n=2), friend (n=2), or child (n=1). Caregivers had provided care for an average of 8.9 years (SD=7.8; range = <1 year to 25 years) with an average of 10.2 hours per week of care provided (SD=9.9; range=1–25 hours).

    Table 1

    Sociodemographic characteristics of 13 nominal group technique discussion participants

    NGT discussions: challenge items

    The three groups generated 24, 27 and 38 original caregiving challenge items for a total of 89 (online supplementary appendix S3), although there were duplicate items across groups. Of the 89 original items, 16 items received a mean score for importance between 8.0 and 10.0; 27 items were between 6.0 and 7.9; 39 items were between 4.0 and 5.9; and 7 items were rated <4.0.

    After completion of the groups, the 89 items were distributed to members of the research team and the SSc Caregiver Advisory Team, resulting in the rewording, removal or combining of items. There were 55 unique challenge items remaining after this editing process. Six items were added by the research team and the SSc Caregiver Advisory Committee, resulting in a total of 61 identified challenges (table 2).

    Table 2

    Reduced and categorised list of caregiver generated challenges and item means

    Using a modified set of the categories used in a previous study,22 the 61 challenges were grouped into nine categories (see online supplementary appendix S2 for category definitions). There was 88% agreement of item categorisation between raters. The definitions and rules were used in order to reach a consensus for the placement of each item. As shown in table 2, the nine categories included physical health concerns (n=3 items); financial problems and work or employment problems (n=4 items); role strain (n=8 items); information, resources and support needs (n=15 items); fear, anxiety and uncertainty (n=3 items); general emotional difficulties (n=7 items), emotional difficulties of the care recipient (n=8 items); changes in relationship dynamics with care recipient (n=10 items); and changes in social interactions (n=3 items).

    NGT discussions: support service items

    The three groups generated 13, 13 and 15 original items reflecting preferred support service options (see online supplementary appendix S4). Among the 41 items, 12 received a score reflecting likelihood of using the service between 9.0 and 10.0; 6 between 8.0 and 8.9; 15 between 6.0 and 7.9; and 8 less than 6.0. There was a high degree of duplication of items across groups. Thus, the 41 total items were reduced by the research team to 18 unique items, and these items were reviewed and edited by the research team, as necessary (table 3). These 18 items included support services delivered online (n=7), by telephone (n=2), hard-copy resources (n=2) and in-person delivery (n=7).

    Table 3

    Reduced list of caregiver-generated support services and item means


    In the present study, we used a novel NGT method to identify challenges faced by informal caregivers of persons with SSc and potential support services to address these challenges. Based on the quantity and variety of items that caregivers generated, NGT discussions were effective for developing items for our planned needs assessment survey of SSc caregivers. Use of the NGT enabled caregivers to directly share their perspectives and provided a mechanism for direct caregiver input in the development of survey that will be distributed to a larger sample of caregivers to persons with SSc.

    A final list of 61 survey items that reflect unique challenges related to caregiving in SSc was generated. These challenges included physical health concerns; financial problems and work or employment problems; role strain; information, resources and support needs; fear, anxiety and uncertainty; general emotional difficulties; emotional difficulties of the care recipient; changes in relationship dynamics with the care recipient; and changes in social interactions. The largest number of items reflected challenges related to unmet information, resource and support needs. Caregivers also generated items that reflected 18 support services that could be delivered to help address difficult aspects of caregiving. Support services that were delivered online, by telephone, in-person and through hard-copy resources were all identified by caregivers as being potentially helpful. Online or in-person delivery of support services that focused on providing education and emotional support were the most common suggestions.

    To our knowledge, this is the first study to gather information on challenges from informal caregivers of people living with SSc using NGT discussions. Our findings, however, can be compared with a thesis that included individual interviews with 13 caregivers of persons with SSc.20 Several of the challenges generated during the NGT discussions that related to general emotional difficulties are similar to themes identified in the thesis, including guilt, frustration and stress. Further, financial strain and career adjustments, relational strain and difficulty managing multiple roles were reported as a challenges of caregiving in our NGT discussions and in previously reported interviews.20

    Our results can also be compared with studies of caregivers to persons with more common diseases, including Huntington’s disease, cancer and Alzheimer’s disease. Caregivers to persons with more common diseases have described several similar challenges, including understanding and managing the patient’s medical needs, changes to the patient–caregiver relationship, managing disease-related problems and concerns about the care recipient not receiving adequate care.22–25 Our findings should also be considered in the context of rare diseases. Previous literature has suggested that rare diseases pose additional challenges for both the caregiver and the person living with the disease.17 26 27 Our results are in line with this research as the most commonly reported challenges related to the rarity of the disease and difficulty obtaining necessary information, resources and support.

    There are currently no formal resources in place for SSc caregivers, and no previous research has administered support service interventions among caregivers to persons with SSc. There are, however, many support services that have been developed to lessen the burden associated with caregiving in common diseases,28–31 and some of these may be able to be adapted for SSc caregivers. For example, caregiver support groups, teleconference-based interventions and educational sessions32–36 have been provided for caregivers of persons with dementia, asthma, allogeneic haematopoietic stem cell transplantation, and cancer and were identified in our study as support services that SSc caregivers would be likely to use.

    Strengths and limitations

    The present study has important strengths. For example, no other studies have used NGT discussions as a way of eliciting information from caregivers of people with rare diseases. Applying this methodology allowed us to generate a robust list of potential survey items in the words of stakeholders themselves and provided initial quantitative ratings of the importance of challenges and likelihood of service use for the items generated. Further, our research team included our Scleroderma Caregiver Advisory Team, which comprised caregivers to persons with SSc. This allowed for the incorporation of stakeholder input at each stage of the research and will aid in the facilitation of follow-up survey work and the eventual implementation of the findings. An advantage of the NGT compared with standard focus groups is that focus groups use open-ended questions to solicit discussion, and researchers must then develop survey items from the themes that were discussed. The NGT allowed for direct caregiver input that did not require researchers to interpret and extrapolate from what participants shared in the discussion to generate survey items.

    There are limitations to consider when interpreting the results of this study. First, the NGT discussions were held at national patient conferences. This may have influenced the characteristics of participants and the generalisability of results as only caregivers attending the 2016 American or Canadian conferences were eligible for participation in the groups. Specifically, our sample may have overincluded caregivers who are well connected to the SSc community. Second, participants only rated items one time in the groups. NGT discussions sometimes ask participants to re-score items after discussing the results of the first round of scoring, similar to a Delphi process.37 Participants were only asked to rate items once and ratings were completed independently rather than discussed with other participants. Previous studies, however, have found that when asked participants state that item ratings would generally remain unchanged if asked to rate items a second time.38 Third, participation in the NGT discussions was restricted to caregivers who were literate. Lastly, four challenges were removed from our list of unique items due to the vague content of these items. During the NGT discussions, gathering additional specifications about the challenges may have been helpful to ensure that the items elicited from participants were retained. Despite these limitations, the study provides important information that can be used to inform the development of a survey to determine caregivers’ most important challenges, and the support services that caregivers would be most likely to use.


    The NGT was an efficient approach for gathering caregiver input to aid in the development of survey items. We found that caregivers of persons with SSc face many challenges and have substantial unmet needs. Some of the most important challenges identified were related to information, resources and support needs. A range of possible support services were identified, with caregivers reporting being most likely to use services delivered in-person or through online platforms for education and emotional support, including internet-based psychological and emotional self-help tools. The findings from the present study suggest that programmes offered online may result in a greater likelihood of caregiver participation. The results of the present study will be used to construct a survey that will be disseminated online to a larger number of SSc caregivers in order to better understand the relative importance of the challenges identified and the likelihood of use of possible support services. The results of the present study, combined with survey results, will be used to develop SSc caregiver support services.


    1. 1.
    2. 2.
    3. 3.
    4. 4.
    5. 5.
    6. 6.
    7. 7.
    8. 8.
    9. 9.
    10. 10.
    11. 11.
    12. 12.
    13. 13.
    14. 14.
    15. 15.
    16. 16.
    17. 17.
    18. 18.
    19. 19.
    20. 20.
    21. 21.
    22. 22.
    23. 23.
    24. 24.
    25. 25.
    26. 26.
    27. 27.
    28. 28.
    29. 29.
    30. 30.
    31. 31.
    32. 32.
    33. 33.
    34. 34.
    35. 35.
    36. 36.
    37. 37.
    38. 38.
    View Abstract


    • Contributors DBR and BDT were responsible for the study conception. DBR, STG and BDT contributed to data collection. DBR, MCA, KAT, STG, VLM, MH, BDT and the SCAT contributed to data analysis and interpretation. DBR, MCA and BDT contributed to drafting the manuscript. All authors provided a critical revision of the manuscript and approved the final version of the manuscript. BDT is the guarantor. The authors had full access to the data and take responsibility for the integrity of the data and the accuracy of the data analysis.

    • Funding This research was supported by a grant from the Rare Disease Foundation and the BC Children’s Hospital Foundation (Funding Reference Number 173-16; PI DBR).

    • Competing interests DBR was supported by a Vanier Graduate Scholarship. STG was supported by a Fonds de Recherché Santé Québec Doctoral Award. BDT was supported by a Fonds de recherche du Québec – Santé researcher salary award.

    • Patient consent Obtained.

    • Ethics approval The Research Ethics Committee of the Jewish General Hospital, Montreal, Quebec, Canada.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available.

    • Collaborators Scleroderma Caregiver Advisory Team Members: Marcia Greiten, Goleta, California, USA; Christy McCaffrey, Philadelphia, Pennsylvania, USA; Judi McDonald, Bedford, Nova Scotia,Canada; John Michalski, Macomb, Michigan, USA; Mathieu Ross, Quebec City, Quebec, Canada; Lisa Spinney, Berlin, Massachusetts, USA; Terry Stacey, Oshawa, Ontario, Canada;Linda Tarantino, Larkspur, California, USA.

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.