You are here

Article Text

Article menu

Download PDFPDF

XML
Health services research
Research
User-testing an interactive option grid decision aid for prostate cancer screening: lessons to improve usability
  1. Peter Scalia1,
  2. Marie-Anne Durand1,
  3. Marjan Faber2,
  4. J A Kremer3,
  5. Julia Song1,
  6. Glyn Elwyn1
  1. 1 The Dartmouth Institute for Health Policy and Clinical Practice, Dartmouth College, Hanover, New Hampshire, USA
  2. 2 Radboud University Medical Centre, Radboud University, Nijmegen, The Netherlands
  3. 3 Department of Obstetrics and Gynaecology, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
  1. Correspondence to Peter Scalia; peter.scalia{at}dartmouth.edu

Abstract

Objective To user-test a web-based, interactive Option Grid decision aid ‘prostate-specific antigen (PSA) test: yes or no?’ to determine its usability, acceptability and feasibility with men of high and low health literacy.

Design A semi-structured interview study.

Setting Interviews were conducted at a senior centre, academic hospital or college library in New Hampshire and Vermont.

Participants Individuals over 45 years of age with no history of prostate cancer who voluntarily contacted study authors after viewing local invitations were eligible for inclusion. Twenty interviews were conducted: 10 participants had not completed a college degree, of which eight had low health literacy, and 10 participants had high health literacy.

Intervention An interactive, web-based Option Grid patient decision aid for considering whether or not to have a PSA test.

Results Users with lower health literacy levels were able to understand the content in the tool but were not able to navigate the Option Grid independent of assistance. The tool was used independently by men with high health literacy. In terms of acceptability, the flow of questions and answers embedded in the tool did not seem intuitive to some users who preferred seeing more risk information related to age and family history. Users envisioned that the tool could be feasibly implemented in clinical workflows.

Conclusion Men in our sample with limited health literacy had difficulty navigating the Option Grid, thus suggesting that the tool was not appropriately designed to be usable by all audiences. The information provided in the tool is acceptable, but users preferred to view personalised risk information. Some participants could envision using this tool prior to an encounter in order to facilitate a better dialogue with their clinician.

Ethics approval The study received ethical approval from the Dartmouth College Committee for the Protection of Human Subjects (STUDY00030116).

  • prostate-specific antigen test
  • web-based, interactive decision aids
  • option grid patient decision aids
  • user-testing
  • health literacy
  • think-aloud

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

http://dx.doi.org/10.1136/bmjopen-2018-026748

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • The strength of this study was the use of the think-aloud technique in conjunction with in-depth interviews to examine the usability, acceptability and feasibility of the Option Grid.

    • The sample includes participants with lower levels of health literacy and education who are not computer-savvy.

    • Due to two study authors’ (GE and M-AD) involvement in developing the prostate-specific antigen interactive Option Grid, PS conducted the interviews and analysis of the data to mitigate the risk of bias.

    • A weakness of our study is the fact that we did not use a validated framework to user-test patient decision aids.

    • We recognise that including a more diverse sample of men (ie, ethnicity and geographical location) could have provided us with a different user-testing perspective.

    Background

    A growing number of developers are creating web-based, interactive decision aids and this includes tools for men considering the prostate-specific antigen (PSA) test.1 Patient decision aids are designed to provide users with evidence-based information on a particular health condition or screening test, and many men are using these tools when needing to make a preference sensitive decision – such as whether they should have a PSA test. Part of the development process is to user-test these web-based PSA decision aids to ensure they are easy to use and meet the needs and desires of men.2–4

    The International Patient Decision Aid Standards indicate that users should be involved in the development process of these interventions.2 Elwyn et al proposed a four-step process map for developing web-based tools which recommends usability testing and field testing.3 Once a prototype is deemed ready by developers, it should be user-tested by stakeholders, including patients, to provide their perspective on elements of the tool that need to be improved (ie, navigation and design). This is a key step in the process prior to introducing the web-based decision aid in clinical settings for further evaluation.3 A review conducted by Coulter et al outlined similar processes to evaluate patient decision aids all of which recommend user-testing or determining the needs of patients who will eventually use these tools.4 The Ottawa Decision Support Framework, for example, described an iterative process to decision aid development which includes ‘assessment of determinants of decisions’ through the lens of the patient and provider.4 5 Patient input enables developers to create robust tools that are acceptable to potential users while addressing their needs and concerns.6 However, involving patients is time-consuming and costly.4 Many developers bypass the user-testing step and post tools that may not be suitable for the target audience.

    According to a recent explanatory study by Dugas et al, of the patient decision aid development projects that have been conducted, less than 20 per cent include vulnerable populations (ie, patients with low health literacy).7 Health literacy is defined as ‘the ability to perform basic reading and numerical tasks required to function in the healthcare environment’.8 9 A clear theme regarding web-based tools that have been user-tested by low health literacy populations is that they are helpful (improve knowledge and reduce decisional conflict), but improvements are needed to resolve navigation issues and make them easier to use.10–15 Developers tend to see the opportunity to use online platforms to provide more information than they otherwise would on paper versions, but this increases the cognitive load, particularly for individuals unfamiliar with medical jargon.11 12 Evidence also exists that vulnerable populations may need assistance when using web-based applications.16 Fine-tuning the information elements and layout of these web-based tools to provide simplicity are frequently suggested alterations by low health literacy users regardless of health topic.

    In the case of web-based PSA screening tools, it is clear that the majority available to the public have not been user-tested or tested with men of low health literacy – a patient population that is in need of access to evidence-based information regarding the screening test. We searched for, and found, only one web-based PSA screening tool that has been user-tested. It was designed for men with a family history of prostate cancer who provided positive feedback related to navigation, the amount of information and time required to complete the decision aid.17 However, this tool was customised for a certain segment of the male population and did not take into account the experience of men with lower levels of health literacy. Our study fills this gap by user-testing a web-based interactive tool known as Option Grid patient decision aids, with men of various education and health literacy backgrounds.

    The aim of our study is to user-test the ‘PSA test: yes or no?’ web-based, interactive Option Grid patient decision aid to determine its usability, acceptability and feasibility with men of higher and lower health literacy.

    Methods

    Design

    A semi-structured interview study was designed and reported according to the Consolidated criteria for Reporting Qualitative research checklist using purposive sampling, with embedded think-aloud methods to evaluate the user experience with the ‘PSA test: yes or no?’ interactive Option Grid decision aid. The think-aloud technique (see data collection section) was employed to examine the activities and comments of the user and get feedback on the usability and feasibility of the tool. Semi-structured interviews were conducted following users’ completion of the web-based Option Grid to get a better understanding of how to improve the interactive version.

    Recruitment

    We posted advertisements at bus stops throughout the Upper Valley region describing the study in order to invite men to use the PSA Option Grid and provide feedback. Also, coordinators at community centres and senior centres in the Upper Valley granted us permission to post advertisements at their centres. The advertisements encouraged men who were interested in participating to contact PS to agree on an interview time. The advertisement included information that participants would receive a $20 gas card for their participation.

    Settings

    Interviews were conducted at three locations in the Upper Valley of New Hampshire and Vermont: (i) the Dartmouth-Hitchcock Medical Centre, (ii) Dartmouth College library and (iii) the Upper Valley Senior Centre in Lebanon, New Hampshire.

    Participants

    Individuals over 45 years of age living in the Upper Valley of New Hampshire and Vermont with no history of prostate cancer, who voluntarily contacted study authors via phone or email, and indicated their willingness to participate, were eligible for inclusion. Participants were excluded if they could not read English. Participants were not excluded if they underwent a PSA test at any point prior to their participation in the study. Study aims and procedures were explained to the participants using an information sheet. Participants were informed, prior to their consent, that any information provided for the study would be kept confidential and stored securely with restricted access, thereby minimising risks to the privacy of those involved in the study.

    Patient and public involvement

    The development of the research question, study design and procedure are employed to improve the patient experience using a web-based patient decision aid like Option Grid. Patients were not involved in designing or conducting the study. Results were not disseminated to study participants.

    Development of the PSA option grid decision aid

    Option Grid patient decision aids are available in both a paper-based and web-based, interactive version and provide brief, evidence-based information on the available treatment options for various health conditions or screening tests, including the PSA.18 An editorial team comprised of researchers and physicians, and led by GE, developed the web-based, interactive PSA Option Grid based on the International Patient Decision Aid Standards. The team, in consultation with patients, determined the frequently asked questions (FAQs). Evidence-based answers to those FAQs were provided by conducting a literature review. The search focused on systematic review papers to determine the risk information embedded in the tool. A document listing the references used to formulate the answers for each FAQ accompanies the interactive PSA Option Grid. Users have the option of printing a paper-based version of the tool or using the interactive, web-based version of the PSA Option Grid. Printed Option Grid decision aids use a tabular format to identify the available treatment options for a chosen healthcare topic (columns) and the questions that patients most frequently ask (rows).18 The content embedded in the tool consists of the evidence-based answers to the FAQs.19

    Users who select the interactive version are asked to provide demographical information such as their role (patient/patient caregiver or healthcare professional), race, age group, gender, ethnicity and geographical region. Users are asked: ‘Do you already have a preferred option in mind?’ Users can select one of three options: having a PSA test, not having a PSA test or not sure. They are then asked how strong their preference is – weak, moderate or strong. Next, users fill out a four-item, validated SURE (Sure of myself; Understand information; Risk-benefit ratio; Encouragement) survey which helps measure their level of decisional conflict,20 21 followed by a series of five questions to check their knowledge of the PSA test. A series of 10 FAQs are then presented, always in the same order. Users select their preference leaning based on the answer given for each treatment option. They then identify, on a scale of 0 (not important at all) to 5 (extremely important), the importance of the information presented. Once all 10 steps are completed, the user indicates their final preference, identifies their preference strength based on the scale and completes the SURE survey and knowledge questions. Finally, the user is presented with a summary of the choices they make for each FAQ.

    Data collection

    All interview sessions were conducted face-to-face with the participant and occurred in a private room at one of the three study settings. The Chew et al health literacy measure was completed by all participants.9 It is a validated three-item measure that uses a Likert scale (always, often, sometimes, occasionally, never) to determine the confidence with which individuals fill out forms and medical forms independent of any help, and how often they require help reading hospital materials.22 The study author queried each participant on their highest level of education attained. For the purposes of this study, participants who did not graduate with a college degree were considered to be of lower educational attainment.

    Participants were provided with a laptop. Participants in this study were not provided the option to view or print the paper-based version of the PSA Option Grid. The interactive PSA Option Grid webpage was open, so participants could begin the interactive journey when ready. As participants were using the interactive tool they were asked to verbalise their thoughts - a technique known as think-aloud. Think-aloud was used because it enabled us to have a better understanding of the usability, feasibility and the impact on user ‘cognition and emotion’.23 24 It provided participants with the opportunity to voice their satisfaction or dissatisfaction with various elements of the interactive Option Grid, recommend changes and identify any difficulties related to navigation or understanding.

    Following the completion of the interactive PSA tool, we conducted semi-structured interviews (interview guide can be viewed in online supplementary appendix 1). Questions (derived from Morville’s framework – see analysis section) were focused on five elements: (1) usability (including design and navigation), (2) usefulness (including questions on content), (3) desirability (positive or negative features), (4) value and (5) credibility. Interviews were audio-recorded and duration of the interviews was noted. Data was collected until saturation was achieved – sampling more data would not have led to new information regarding usability of the tool as users kept repeating the same concerns or ideas. Transcripts were not returned to participants for comment or correction.

    Supplementary file 1

    Analysis

    We searched for, but could not find, a validated framework to user-test a web-based patient decision aid. This is a relatively new field, and although there does not seem to be a well-used or recognised framework, we opted to use Morville’s ‘honeycomb’ framework to provide structure to our data (see figure 1).25 It has been used in the past to test oral health education applications, mobile device applications, business process management tools and social software.26–29 This framework has also been employed once before to user-test a decision aid known as the ‘decision-box’, and by the Cochrane Collaboration to test user’s experience with their systematic review library.30 31 We applied this framework to analyse our data since it contains all the features that we wanted to evaluate in our interviews. It was developed to assess seven different elements associated with the user’s experience with an intervention: usability, usefulness, credibility, desirability, value, findability and accessibility.28 Usability refers to the ease of use, and the degree of satisfaction attained from using the PSA interactive Option Grid. Usefulness relates to the value of the tool to the user’s knowledge-base or potential decision-making regarding the PSA test. Credibility explores the user’s level of trust for the evidence-based information embedded in the tool. We used the desirability element of the framework to examine the type of changes the user recommends making to improve the user experience. Value explores whether the interactive PSA Option Grid ‘advances the mission’, which in this case, means improving knowledge of the PSA test so a user can make a more informed decision, or can have a more efficient conversation with their provider in future encounters. For the purpose of this study, findability and accessibility were not assessed because study authors introduced the interactive PSA Option Grid to the user. We did not test the ease with which one could find the interactive tool online, or the barriers to access. We applied five of the seven (usability, usefulness, credibility, desirability, value) ‘honeycombs’ to evaluate any issues the user may encounter with the PSA interactive Option Grid which may be the basis for re-design.

    Figure 1
    Figure 1

    Morville’s user-experience ‘honeycomb’ framework.

    Qualitative data analysis

    We employed Wallace et al’s recommendation to use the ‘sometimes’ response on Chew’s Likert scale as the cut-off to determine the number of participants with ‘limited or marginal health literacy skills’.32

    User-testing sessions were recorded and transcribed by PS. The framework analysis method was used to analyse and code the data.33 We conducted a deductive thematic analysis using Morville’s framework, and the data was coded independently by two researchers (PS and JS). We used this type of qualitative analysis because the coding and theme development were directed by the framework which enabled us to best achieve our study objectives. The ATLAS.ti software was used to assign codes, generated based on the five facets of the framework, to sections of participant statements, and additional codes were created to group codes together. All codes were reviewed to determine the themes associated with usability and feasibility of the PSA interactive Option Grid. We also highlighted specific changes or alterations suggested by the participant to improve the tool.

    Results

    Twenty interviews were conducted with men in the Upper Valley region. Ten participants interviewed had not completed a college degree (low educational attainment) and 10 participants completed at least one college degree (high educational attainment). Of the 10 participants in the low educational attainment group, eight had low health literacy skills. All participants who achieved a college degree had high health literacy. All participants were white and their ages ranged from 49 to 81. See table 1 for details.

    View this table:
    Table 1

    Participant characteristics

    Thematic analysis

    We identified four key themes: (1) Perceived usability was contingent on familiarity with digital interfaces (and on health literacy), (2) Desire for revisions to improve user experience, (3) The value of communicating risks and tailoring content to individual attributes like age and family history and (4) Credible source of information, useful for pre-visit use. See table 2 for themes and sub-themes.

    View this table:
    Table 2

    Themes and sub-themes derived from user-testing data

    Theme 1: perceived usability was contingent on familiarity with digital interfaces (and on health literacy)

    According to Morville’s framework, usability encompasses the ease of use, and the degree of satisfaction attained from using the intervention. Throughout their journey, participants with lower levels of health literacy (according to the Chew health literacy measure) described their difficulty with using and navigating the tool which affected their level of satisfaction with the intervention. Participants with higher health literacy levels used the tool with ease but envisioned potential challenges for men with lower levels of education.

    Six of the eight users who had lower levels of health literacy and low educational attainment were unable to use or navigate the PSA interactive Option Grid due to their stated unfamiliarity with the computer interface. The interviewer (PS) had to read and select the various options for the participant. Using an electronic device proved to be an intimidating factor in their user-testing experience and decreased satisfaction:

    I’m not computer literate. That’s obvious…I’m intimidated by computers to be honest. (Participant 2 – age 70, low educational attainment and health literacy)

    I don’t like this. You go ahead and use it. (Participant 3 - age 70 to 75, low educational attainment and health literacy)

    According to multiple users, the majority of the population at the senior centre would not be computer literate. We were advised to simplify the tool for a lower health literacy population:

    I’d say about 40% of the people that come here are computer literate. But, for older people that aren’t, and I’m thinking about people that come here…you need a more simplified version. (Participant 5 – age 76, low educational attainment and health literacy)

    The men with higher health literacy, who were also part of the high-education bracket, had no issues with using or navigating the tool, but did express their concern for men who lack a higher level of education. They could envision how this interactive tool may not be usable for those men and hypothesised that a coach would be needed to guide them through the tool.

    I think they [participants with low educational attainment] would need a coach to do this. (Participant 19 - age 65, high educational attainment and high health literacy)

    Lastly, twelve users recommended we increase the size of the words to make the content easier to read for all users. Regardless of health literacy levels, elderly men being the target population of the PSA Option Grid have specific needs in terms of usability:

    The font needs to be bigger. Particularly for people like me who may have vision problems… everything needs to be bigger. (Participant 3 - age 70 to 75, low educational attainment and health literacy)

    The discrepancy between men with higher and lower health literacy levels regarding usability was evident and is a factor in the ability to use and navigate the PSA interactive Option Grid.

    Theme 2: desire for revisions to improve user experience

    The desirability component of the framework highlights the desire to change various attributes of the tool to improve the user experience. What type of changes do users want to see? We describe these changes in detail.

    Participants had some difficulty understanding the actions they needed to take when faced with the FAQ portion of the interactive tool. For instance, the first FAQ poses the question: what does the test involve? The having a PSA test option states that the PSA is a blood test that measures the antigen level in the blood from the prostate gland, and the not having a PSA test option simply states ‘does not apply.’ For the second FAQ, the same information was associated with each option – ‘15% of American men will develop prostate cancer in their lifetime’. Eight participants felt like the information associated with each option were unclear, leading to confusion on which screening option they should choose.

    So, I’m confused as to why I have to check this box as opposed to this box because they both have the same information. What are you trying to tease out here in this question with the two choices? (Participant 12 - age 69, high educational attainment and health literacy)

    To alleviate confusion, users suggested we alter the organisation of the material for each FAQ. Participants indicated we should state a fact related to the PSA test, and then ask: does this information make you more or less inclined to have, or not have, a PSA test? Then, the participant can check the box based on their preference. This adjustment would also decrease the amount of material embedded in each FAQ page.

    Start out with this bit of information up here, and then that’s when you get the two options. Put the statement you just made, and then add…does this information make you more or less inclined to have the test or not? (Participant 10 - age 66, high educational attainment and health literacy)

    I’m suggesting that you have the piece of information, and then you’re given two boxes, yes/no, click one. (Participant 12 - age 69, high educational attainment and health literacy)

    I would be more…it would seem a little more logical to me to have a number of those facts laid out, and then say: do these make you feel more or less likely to want to have a PSA? (Participant 18 - age 65 to 70, high educational attainment and health literacy)

    For some FAQs, the having a PSA test option provided risk information. Below that risk information was a ‘data viewer’ option that, if clicked, provided users with an icon array to help them better understand the frequencies or percentages presented (see figure 2). Only two participants in the sample recognised the data viewer due to its small presence on the page. Once we directed their attention to the icon array, users indicated that it should be embedded with the rest of the information:

    Figure 2
    Figure 2

    Icon array embedded in the ‘data viewer’ option in the PSA interactive Option Grid. PSA, prostate-specific antigen.

    Yeah, pictorials are good. I didn’t have problems with the 15%, but I would incorporate this (the icon array) in here (on the FAQ page). (Participant 15 - age 70, high educational attainment and health literacy)

    The third recommended adjustment was the removal of the importance ratings at the bottom of each FAQ. The majority of users did not understand the purpose or usefulness of that exercise, considering their attention was focused mainly on the information. The information was generally considered to always be of high importance to the user:

    So, to me the importance ratings of each question were less important than the stuff above. I thought the top part was informative, and then the zero to five scale was informative to you, the researcher, but not to me. (Participant 8 - age 50 to 55, high educational attainment and health literacy)

    Ultimately, users suggested to re-organise the material for each FAQ by stating a piece of information, and then asking whether or not that information made the user more or less inclined to have the PSA test, thus clarifying the action required. Moving the icon array embedded in the data viewer to accompany the information for each FAQ would also help users visualise and understand the percentages and frequencies presented.

    Theme 3: the value of communicating risks and tailoring content to individual attributes like age and family history

    Value, according to the framework, refers to the attributes that participants deem most valuable. Viewing the tool through this lens, we identified that the risk information was most valued. In particular, users wanted to see prostate cancer risk information related to age and family history.

    Participants felt that the greatest asset of the PSA interactive Option Grid was the level of high-quality, evidence-based information. It was also easy to understand, and much of the numerical information was new to users:

    There were a couple of things that I wasn’t aware of regarding the PSA test, which came up in here. About 30% of men with a high PSA have prostate cancer. I didn’t know that. (Participant 2 – age 70, low educational attainment and health literacy)

    Even for participants who already possessed knowledge of the risks and benefits associated with the PSA test, they felt that the tool provided the foundation for a discussion with a physician:

    I think this is a good introduction because you have people out here who haven’t considered or worried about it, and need to be educated. They need to be brought up to speed… as an introduction this will work. You may have had some people today that didn’t know what the heck they were doing or haven’t really paid attention… (Participant 3 - age 70 to 75, low educational attainment and health literacy)

    In particular, participants highlighted the value of the various probabilities associated with the PSA test, including: likelihood of developing prostate cancer in one’s lifetime, the chances of getting prostate cancer if the PSA test is high or normal, the percentage of prostate cancers that are aggressive and the risks associated with prostate biopsies and cancer treatments.

    The immediate risk of the various procedures, or if you had a high PSA, the risk associated with having a biopsy. That was the most helpful piece of information for me. (Participant 9 - age 49, high educational attainment and health literacy)

    Despite the participants’ satisfaction with the true content in the tool, they indicated that there were two key information elements missing. The first is age-specific risk information related to prostate cancer. Stratifying the risk information according to age would help men better understand when the ideal time is to consider the PSA test. Second, how does family history influence one’s risk of prostate cancer? How important is family history for those considering the PSA test? These are questions that participants felt should be addressed in order to create a more comprehensive tool for users.

    I think it would be interesting to know something about age because I’ve been told, I heard, that if you are over a certain age, the speed at which the average prostate cancer moves is very slow that you’ll probably die of something else first. (Participant 13 – age 75, high educational attainment and health literacy)

    Theme 4: credible source of information, useful for pre-visit use

    According to the framework, credibility refers to the trust given to the tool’s contents. It is because of the trust in the evidence-based nature of the PSA Option Grid that participants found the tool useful. The intervention was useful in expanding the user’s knowledge-base and helped him make a decision regarding the PSA test. The two mechanisms of the framework are linked in the sense that the tool represented a credible source of information for users, and it could be used in a clinical setting prior to their discussion with the physician.

    Participants felt that the ideal time to complete the tool would be prior to the clinical encounter due to the time pressures physicians experience. The patient can then bring the summary of their results to the physician to facilitate a better discussion about whether or not the PSA test is worthwhile.

    Yes. I think people could fill this thing out before the visit, so that they don’t have to sit with the doctor and hash through it. (Participant 8 - age 50 to 55, high educational attainment and health literacy)

    Participants recognised that physicians have limited time with their patients, but the need to have a more constructive discussion about the PSA test still exists. They believed that the interactive PSA Option Grid represented the basis for an improved interaction with the provider because it was a credible source of information. There is a plethora of resources available in print or online regarding the PSA test, but men indicated their difficulty deciphering which information is credible and which information is not. All participants felt that being provided with one source that can provide accurate facts can be helpful to men in their decision-making, alleviating concerns of getting misinformation.

    Everyone Google’s…all the time, and so, more and more people, are going to their physician with questions they found online…I feel like a tool like this if presented within the context of a healthy conversation with your physician is valuable. (Participant 9 - age 49, high educational attainment and health literacy)

    Discussion

    Overall, men indicated that the interactive PSA Option Grid was useful despite the challenges some faced engaging with the tool, the information acceptable with an emphasis to add more risk information associated with age and family history and potentially feasible to implement this tool prior to a clinical consultation. The usability issues for participants with low health literacy can perhaps be attributed to an interactive tool that was not designed for such a vulnerable population. It is worth noting that many men with lower health literacy also seemed unfamiliar with computers and reluctant to use the online intervention independently. The difficulty experienced in navigating the digital interface, feelings of intimidation with having to use a laptop, preference for a paper-based version and desire for changes to the layout and presentation of information all suggested that the PSA Option Grid had not paid enough attention to usability. The risk information provided by the Option Grid was found to be highly valuable with most hoping to see content tailored to their specific attributes like age and family history in a future version of the tool. The PSA Option Grid also represented a credible source of information, and users envisioned using this tool prior to the clinical encounter to facilitate an improved discussion with their physician.

    The strength of this study was the use of the think-aloud technique in conjunction with in-depth interviews to examine the usability, acceptability and feasibility of the tool. The purposive sampling method enabled us to include participants across the health literacy spectrum, including those with lower levels of health literacy and education who are not computer-savvy. Due to two study authors’ (GE and M-AD) involvement in developing the PSA interactive Option Grid, PS conducted the interviews and analysis of the data to mitigate the risk of bias. A weakness of our study is the fact that we did not use a validated framework to user-test patient decision aids. Also, we did not provide the option for participants to use a tablet or smartphone to access the PSA Option Grid. It may be that using these devices could have changed some of our results. We also recognise that interviewing a more heterogenous sample that includes men of different ethnicities and from different locations from across the USA would have provided us with a different user-testing perspective and made our results more applicable to decision aid developers across cultural contexts.

    Our research highlights the importance of user-testing interactive decision aids prior to making it available to the online world. Developing a decision aid that is usable by men with lower health literacy involves an iterative interview process as described by Barton et al in their development of an encounter-based decision support intervention for rheumatoid arthritis medications.34 Patients were interviewed on two separate occasions in order to refine the tool prior to pilot testing.34 The developers of the PSA Option Grid did not user-test prior to its publication and based on this study we know men with lower health literacy struggled to navigate the application. Using a laptop proved to be a challenging exercise that provoked feelings of intimidation - an unprecedented finding that adds a layer of complexity for developers of interactive tools. The design changes recommended by users also suggests a desire for simplicity by embedding one fact about the PSA per FAQ to assess preference. Developers of decision aids should user-test their tools to ensure the layout and presentation of information meets the needs of low health literacy patients, and so that they can navigate the tool without assistance.

    Our study indicates that men value risk information regarding the PSA test, and seek information tailored to their own characteristics, particularly age and family history. Survey and qualitative research echo these findings.35 36 Despite information needs being highly variable and dependent on demographics, the desire for risk information remained stable for men making a decision on whether to have the PSA test or not.35 Davison et al concluded that ‘men who had sons, a positive family history and lower levels of education ranked hereditary risk’ as a key component in their prostate screening decision-making process.37 Family history was repeatedly mentioned by participants in our sample as a factor that weighs heavily on the decision-making process. Although sexual function was not an issue raised by participants in our study, research shows that the potential effects of prostate screening on sexual function is significant to younger men considering the PSA test.38 39 In particular, participants with higher education consider sexual function to be of value when reflecting on the test.39

    The source of the various facets of information regarding the PSA test is also a major theme in the literature. The internet has become the prime resource for men seeking credible information on the PSA test.40 The preferred sites are those that are recognised or ‘institutions with established reputations’.41 Also, men often resort to online patient discussion forums for information support – a platform that was not mentioned by participants in our sample.42 43

    Participants felt that it would be feasible to use the tool prior to the consultation with the physician. According to the Ottawa Inventory of patient decision aids, there are eight PSA tools available for men – three of which are designed for use in the encounter. The feasibility or routine implementation of these decision aids has yet to be studied. Future work should examine how and when to best integrate these online decision support tools that have shown to increase patient knowledge and satisfaction related to the prostate cancer screening decision.44 In addition, the community of decision aid developers should set an agenda to explore how to make web-based tools accessible and usable for low literacy individuals.

    Conclusion

    The PSA interactive Option Grid decision aid was useful but not optimally designed to ensure usability for all men in the sample. Research is needed to determine how best to increase access to patient decision aids (and the information embedded in these tools) so individuals of all literacy, health literacy and computer literacy levels can benefit. The value of the evidence-based risk information embedded in the tool rendered it credible and acceptable by users who wish to see information associated with family history and age in a future version of the PSA Option Grid. Users envisioned the tool being feasibly implemented prior to their visit with the physician. Overall, this highlights the importance of user-testing web-based decision aids prior to their release in order to get a comprehensive understanding of user needs and priorities. Based on our results, decision aid developers can ascertain that users prefer simplicity regarding the layout and presentation of evidence-based information and prefer to view risk data that is based on their personal attributes (ie, age and family history). However, more research is needed to better understand how interactive, web-based decision aids can be better designed to be usable by individuals across the health literacy spectrum.

    Acknowledgments

    We would like to thank the senior centers in the Upper Valley who enabled us to interview individuals who visit their centre. We are grateful to all participants for their time and effort in helping us improve the online, interactive PSA Option Grid. Lastly, we would like to thank Dr Holly Witteman for her guidance on user-testing methods.

    References

    1. 1.↵
      2. Syrowatka A ,
      3. Krömker D ,
      4. Meguerditchian AN , et al
      . Features of computer-based decision aids: Systematic review, thematic synthesis, and meta-analyses. J Med Internet Res 2016;18:e20.doi:10.2196/jmir.4982
      OpenUrl
    2. 2.↵
      2. Elwyn G ,
      3. O’Connor AM ,
      4. Bennett C , et al
      . Assessing the quality of decision support technologies using the International Patient Decision Aid Standards instrument (IPDASi). PLoS One 2009;4:3:e4705.doi:10.1371/journal.pone.0004705
      OpenUrlCrossRefPubMed
    3. 3.↵
      2. Elwyn G ,
      3. Kreuwel I ,
      4. Durand MA , et al
      . How to develop web-based decision support interventions for patients: a process map. Patient Educ Couns 2011;82:260–5.doi:10.1016/j.pec.2010.04.034
      OpenUrlCrossRefPubMed
    4. 4.↵
      2. Coulter A ,
      3. Stilwell D ,
      4. Kryworuchko J , et al
      . A systematic development process for patient decision aids. BMC Med Inform Decis Mak 2013;13:2.doi:10.1186/1472-6947-13-S2-S2
      OpenUrlCrossRefPubMed
    5. 5.↵
      2. O’Connor AM ,
      3. Drake ER ,
      4. Fiset V , et al
      . The Ottawa patient decision aids. Eff Clin Pract 1999;2:4.
      OpenUrl
    6. 6.↵
      2. Witteman HO ,
      3. Dansokho SC ,
      4. Colquhoun H , et al
      . User-centered design and the development of patient decision aids: protocol for a systematic review. Syst Rev 2015;4:11.doi:10.1186/2046-4053-4-11
      OpenUrl
    7. 7.↵
      2. Dugas M ,
      3. Trottier MÈ ,
      4. Chipenda Dansokho S , et al
      . Involving members of vulnerable populations in the development of patient decision aids: a mixed methods sequential explanatory study. BMC Med Inform Decis Mak 2017;17:1.doi:10.1186/s12911-016-0399-8
      OpenUrl
    8. 8.↵
      2. Chew LD ,
      3. Bradley KA ,
      4. Boyko EJ
      . Brief questions to identify patients with inadequate health literacy. Fam Med 2004;36:12.
      OpenUrl
    9. 9.↵
      Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association. Health Literacy: Report of the Council on Scientific Affairs. JAMA 1999;281:552–7.doi:10.1001/jama.281.6.552
      OpenUrlCrossRefPubMedWeb of Science
    10. 10.↵
      2. Coe AM ,
      3. Ueng W ,
      4. Vargas JM , et al
      . Usability testing of a web-based decision aid for breast cancer risk assessment among multi-ethnic women. InAMIA Annual Symposium Proceedings 2016 (Vol. 2016, p. 411). American Medical Informatics Association.
    11. 11.↵
      2. Smith SK ,
      3. Trevena L ,
      4. Nutbeam D , et al
      . Information needs and preferences of low and high literacy consumers for decisions about colorectal cancer screening: utilizing a linguistic model. Health Expect 2008;11:123–36.doi:10.1111/j.1369-7625.2008.00489.x
      OpenUrlCrossRefPubMedWeb of Science
    12. 12.↵
      2. Holmes-Rovner M ,
      3. Stableford S ,
      4. Fagerlin A , et al
      . Evidence-based patient choice: a prostate cancer decision aid in plain language. BMC Med Inform Decis Mak 2005;5:1.doi:10.1186/1472-6947-5-16
      OpenUrlCrossRefPubMed
    13. 13.↵
      2. Emmett CL ,
      3. Murphy DJ ,
      4. Patel RR , et al
      . Decision-making about mode of delivery after previous caesarean section: development and piloting of two computer-based decision aids. Health Expect 2007;10:161–72.doi:10.1111/j.1369-7625.2006.00429.x
      OpenUrlCrossRefPubMedWeb of Science
    14. 14.↵
      2. Heinrich E ,
      3. de Nooijer J ,
      4. Schaper NC , et al
      . Evaluation of the web-based Diabetes Interactive Education Programme (DIEP) for patients with type 2 diabetes. Patient Educ Couns 2012;86:172–8.doi:10.1016/j.pec.2011.04.032
      OpenUrlPubMed
    15. 15.↵
      2. Bass SB ,
      3. Gordon TF ,
      4. Ruzek SB , et al
      . Developing a computer touch-screen interactive colorectal screening decision aid for a low-literacy African American population: lessons learned. Health Promot Pract 2013;14:589–98.doi:10.1177/1524839912463394
      OpenUrlCrossRefPubMed
    16. 16.↵
      2. Miller DP ,
      3. Weaver KE ,
      4. Case LD , et al
      . Usability of a novel mobile health ipad app by vulnerable populations. JMIR Mhealth Uhealth 2017;5:e43.doi:10.2196/mhealth.7268
      OpenUrl
    17. 17.↵
      2. Wakefield CE ,
      3. Watts KJ ,
      4. Meiser B , et al
      . Development and pilot testing of an online screening decision aid for men with a family history of prostate cancer. Patient Educ Couns 2011;83:64–72.doi:10.1016/j.pec.2010.05.025
      OpenUrlCrossRefPubMed
    18. 18.↵
      2. Elwyn G ,
      3. Lloyd A ,
      4. Joseph-Williams N , et al
      . Option Grids: Shared decision making made easier. Patient Educ Couns 2013;90:207–12.doi:10.1016/j.pec.2012.06.036
      OpenUrlCrossRefPubMed
    19. 19.↵
      2. Fay M ,
      3. Grande SW ,
      4. Donnelly K , et al
      . Using Option Grids: steps toward shared decision-making for neonatal circumcision. Patient Educ Couns 2016;99:236–42.doi:10.1016/j.pec.2015.08.025
      OpenUrl
    20. 20.↵
      2. Légaré F ,
      3. Kearing S ,
      4. Clay K , et al
      . Are you SURE? Assessing patient decisional conflict with a 4-item screening test. Canadian family physician 2010;56:8:e308–14.
      OpenUrl
    21. 21.↵
      2. Ferron Parayre A ,
      3. Labrecque M ,
      4. Rousseau M , et al
      . Validation of SURE, a four-item clinical checklist for detecting decisional conflict in patients. Med Decis Making 2014;34:54–62.doi:10.1177/0272989X13491463
      OpenUrlCrossRefPubMed
    22. 22.↵
      2. Chew LD ,
      3. Griffin JM ,
      4. Partin MR , et al
      . Validation of screening questions for limited health literacy in a large VA outpatient population. J Gen Intern Med 2008;23:561–6.doi:10.1007/s11606-008-0520-5
      OpenUrlCrossRefPubMedWeb of Science
    23. 23.↵
      2. Durand M-A ,
      3. Alam S ,
      4. Grande SW , et al
      . ‘Much clearer with pictures’: using community-based participatory research to design and test a Picture Option Grid for underserved patients with breast cancer. BMJ Open 2016;6:2.doi:10.1136/bmjopen-2015-010008
    24. 24.↵
      2. Lundgrén-Laine H ,
      3. Salanterä S
      . Think-aloud technique and protocol analysis in clinical decision-making research. Qual Health Res 2010;20:565–75.doi:10.1177/1049732309354278
      OpenUrlPubMed
    25. 25.↵
      User experience Design. http://www.semanticstudios.com/publications/semantics/000029.php. (Accessed 15 Dec 2017).
    26. 26.↵
      2. An J
      . Usability testing of oral health education applications – based on a comparison of dentify and toothfix. Journal of Korea Design Forum 2015;49:203–16.
      OpenUrl
    27. 27.↵
      2. Van Greunen D ,
      3. Van Der Merwe A ,
      4. Kotze P
      . Factors influencing BPM tools: The influence on user experience and user interfaces. International Journal of Computing and ICT Research 2010;4:1.
      OpenUrl
    28. 28.↵
      2. LIU T-ping ,
      3. WU X-yun ,
      4. SUN P , et al
      . Frameworks for Exploring the User Experience of Mobile Apps: DEStech Transactions on Environment, Energy and Earth Science, 2016. seeie.
    29. 29.↵
      2. Vuong L
      . Enhancing smartphone’s usability for elderly in Finland: Bachelor Thesis, 2014.
    30. 30.↵
      2. Giguere A ,
      3. Legare F ,
      4. Grad R , et al
      . Developing and user-testing Decision boxes to facilitate shared decision making in primary care--a study protocol. BMC Med Inform Decis Mak 2011;11:11–1.doi:10.1186/1472-6947-11-17
      OpenUrlPubMed
    31. 31.↵
      2. Rosenbaum SE ,
      3. Glenton C ,
      4. Cracknell J
      . User experiences of evidence-based online resources for health professionals: user testing of The Cochrane Library. BMC Med Inform Decis Mak 2008;8:1.doi:10.1186/1472-6947-8-34
      OpenUrlCrossRefPubMed
    32. 32.↵
      2. Wallace LS ,
      3. Rogers ES ,
      4. Roskos SE , et al
      . Brief report: screening items to identify patients with limited health literacy skills. J Gen Intern Med 2006;21:874–7.doi:10.1111/j.1525-1497.2006.00532.x
      OpenUrlCrossRefPubMedWeb of Science
    33. 33.↵
      2. Gale NK ,
      3. Heath G ,
      4. Cameron E , et al
      . Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117.doi:10.1186/1471-2288-13-117
      OpenUrlCrossRefPubMed
    34. 34.↵
      2. Barton JL ,
      3. Koenig CJ ,
      4. Evans-Young G , et al
      . The design of a low literacy decision aid about rheumatoid arthritis medications developed in three languages for use during the clinical encounter. BMC Med Inform Decis Mak 2014;14:104.doi:10.1186/s12911-014-0104-8
      OpenUrl
    35. 35.↵
      2. Feldman-Stewart D ,
      3. Brennenstuhl S ,
      4. Brundage MD , et al
      . Overall information needs of early-stage prostate cancer patients over a decade: highly variable and remarkably stable. Supportive Care in Cancer 2009;17:429–35.doi:10.1007/s00520-008-0514-1
      OpenUrlPubMed
    36. 36.↵
      2. Feldman-Stewart D ,
      3. Capirci C ,
      4. Brennenstuhl S , et al
      . Information needs of early-stage prostate cancer patients: a comparison of nine countries. Radiother Oncol 2010;94:328–33.doi:10.1016/j.radonc.2009.12.038
      OpenUrlCrossRefPubMedWeb of Science
    37. 37.↵
      2. Davison BJ ,
      3. Gleave ME ,
      4. Goldenberg SL , et al
      . Assessing information and decision preferences of men with prostate cancer and their partners. Cancer Nurs 2002;25:42–9.doi:10.1097/00002820-200202000-00009
      OpenUrlCrossRefPubMedWeb of Science
    38. 38.↵
      2. Davison BJ ,
      3. Keyes M ,
      4. Elliott S , et al
      . Preferences for sexual information resources in patients treated for early-stage prostate cancer with either radical prostatectomy or brachytherapy. BJU Int 2004;93:965–9.doi:10.1111/j.1464-410X.2003.04761.x
      OpenUrlCrossRefPubMed
    39. 39.↵
      2. Sidana A ,
      3. Hernandez DJ ,
      4. Feng Z , et al
      . Treatment decision-making for localized prostate cancer: What younger men choose and why. Prostate 2012;72:58–64.doi:10.1002/pros.21406
      OpenUrlCrossRefPubMedWeb of Science
    40. 40.↵
      2. Maliski SL ,
      3. Connor S ,
      4. Fink A , et al
      . Information desired and acquired by men with prostate cancer: data from ethnic focus groups. Health Educ Behav 2006;33:393–409.doi:10.1177/1090198105276223
      OpenUrlCrossRefPubMed
    41. 41.↵
      2. Rozmovits L ,
      3. Ziebland S
      . What do patients with prostate or breast cancer want from an Internet site? A qualitative study of information needs. Patient Educ Couns 2004;53:57–64.doi:10.1016/S0738-3991(03)00116-2
      OpenUrlCrossRefPubMedWeb of Science
    42. 42.↵
      2. Gooden RJ ,
      3. Winefield HR
      . Breast and prostate cancer online discussion boards: a thematic analysis of gender differences and similarities. J Health Psychol 2007;12:1.doi:10.1177/1359105307071744
      OpenUrl
    43. 43.↵
      2. Blank TO ,
      3. Schmidt SD ,
      4. Vangsness SA , et al
      . Differences among breast and prostate cancer online support groups. Comput Human Behav 2010;26:1400–4.doi:10.1016/j.chb.2010.04.016
      OpenUrl
    44. 44.↵
      2. Salonen A ,
      3. Ryhänen AM ,
      4. Leino-Kilpi H
      . Educational benefits of Internet and computer-based programmes for prostate cancer patients: a systematic review. Patient Educ Couns 2014;94:10–19.doi:10.1016/j.pec.2013.08.022
      OpenUrl

    Footnotes

    • Contributors PS, GE, M-AD, MF and JAK designed this qualitative study. PS designed the semi-structured interview schedule, conducted the interviews, analysed the data and wrote the manuscript. JS analysed the data. All authors contributed to the manuscript and approved the final version.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests The Option Grid version that was shared in this work is no longer available. Interactive Option Grid patient decision aids are now available at https://health.ebsco.com/products/option-grid and produced by EBSCO Health. Past: Glyn Elwyn has edited and published books that provide royalties on sales by the publishers: the books include Shared Decision Making (Oxford University Press) and Groups (Radcliffe Press). He has in the past provided consultancy for organizations, including: 1) Emmi Solutions LLC who developed patient decision support tools; 2) National Quality Forum on the certification of decision support tools; 3) Washington State Health Department on the certification of decision support tools; 4) SciMentum LLC, Amsterdam (workshops for shared decision making). Current: Founder and Director of &think LLC which owns the registered trademark for Option Grids TM patient decision aids. Founder and Director of SHARPNETWORK LLC, a provider of training for shared decision making. He provides consultancy in the domain of shared decision making and patient decision aids to: 1) Access Community Health Network, Chicago (Federally Qualified Medical Centers); 2) EBSCO Health Option Grids TM patient decision aids. Non-financial: Glyn Elwyn’s academic interests are focused on shared decision making and coproduction. He owns copyright in measures of shared decision making and care integration, namely collaboRATE, integRATE, Observer OPTION-5 and Observer OPTION-12. These measures are freely available for use by researchers. Dr Marie-Anne Durand is a consultant to Access Community Health Network. Together with Professor Elwyn, she has developed the Option GridTM patient decision aids, which are licensed to EBSCO Health. She receives consulting income from EBSCO Health and may receive royalties in the future.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The dataset used and/or analysed during the current study are available from the corresponding author on reasonable request.

    • Patient consent for publication Not required.