Article Text

Download PDFPDF

Pre-post, mixed-methods feasibility study of the WorkingWell mobile support tool for individuals with serious mental illness in the USA: a pilot study protocol
  1. Joanne Nicholson Ph.D.,
  2. Spenser M Wright,
  3. Alyssa M Carlisle
    1. Department of Psychiatry, The Geisel School of Medicine at Dartmouth, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire, USA
    1. Correspondence to Professor Joanne Nicholson Ph.D.; Joanne.Nicholson{at}


    Introduction Successful competitive employment has been found to be related to enhanced self-esteem, higher quality of life and reduced mental health service use for individuals living with serious mental illnesses (SMIs) including schizophrenia, bipolar disorder and major depression. The effectiveness of the individual placement and support model has been demonstrated in multiple randomised controlled trials in many countries. The management of stress, depression and anxiety in the workplace may be effectively enhanced through digital mental health interventions. The WorkingWell mobile support tool (‘app’) is specifically designed to meet the need for illness management support for individuals with SMI in the workplace, as an adjunct to professional treatment.

    Methods and analysis The WorkingWell app, grounded in evidence-based supported employment, is informed by user experience design. It will be tested in a pre-post design, mixed-methods pilot study to explore issues of feasibility, acceptability and usefulness, and to provide preliminary data on the impact of use. Putative mediators of improved job tenure and psychological well-being, including postintervention changes in social support, self-efficacy and work-related motivation, will be investigated. Forty individuals at least 18 years of age, meeting the eligibility requirements for supported employment services (ie, diagnosed with a mental illness meeting the criteria for severity, duration and treatment), working a minimum of 10 hours per week at study enrolment, and speaking, reading and writing in English will be recruited for the pilot study. Research staff will recruit individuals at community-based mental health agencies; provide orientation to the study, the study smartphones and the WorkingWell app; conduct research interviews including standardised measures as well as semistructured items; and provide technical assistance in telephone calls and inperson meetings. A sample of 10 agency staff will be recruited to obtain further information on the feasibility, acceptability and usefulness of WorkingWell.

    Ethics and dissemination The study design and procedures are approved by the Dartmouth-Hitchcock Medical Center Committee for the Protection of Human Subjects, the Massachusetts Department of Mental Health Central Office Research Review Committee and the Vermont Agency of Human Services Institutional Review Board. Study findings will be disseminated to agency partners, state agencies and funders, and to the research and technology development communities. Findings from the study will inform the design, data collection procedures and protocol for future full-scale randomised controlled trial testing of the effectiveness of the WorkingWell app, as well as investigations of work-related variables as mediators of psychological well-being and quality of life for individuals with SMI.

    • world wide web technology
    • adult psychiatry

    This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

    Statistics from

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • The pilot study will provide preliminary data on the feasibility, acceptability, usefulness and impact of the WorkingWell app, as well as provide feedback on the feasibility and usefulness of study measures and procedures.

    • A mixed-methods approach, employing standardised measures as well as semistructured interviews with participants and agency staff, will provide rich data on the impact and experience of using the app, along with recommendations for improvements in the app and research protocol.

    • Given the exploratory, proof-of-concept nature of the study, the sample size will be small, and there will be no comparison or control group in the pilot study design.

    • A larger sample size would allow for stratification of the sample by individual characteristics potentially related to mobile app use and effectiveness, such as specific psychiatric diagnosis, cognitive impairments, or related physical or sensorimotor functioning.

    • A larger sample size or targeted sample would allow for exploration of the feasibility and usefulness of WorkingWell in diverse employment contexts.


    Over nine million people in the USA experience serious mental illness (SMI), such as schizophrenia, bipolar disorder and major depression, in a given year.1–3 People with SMI have, by definition, major illnesses, typically of long duration and with histories of treatment and hospitalisation, that often result in serious functional impairments in various life role domains.2 4 Psychiatric issues are the leading cause of disability in the USA,5–7 and people with SMI form the largest group of Social Security disability beneficiaries.8–10 Fewer than 15% of people with SMI within the public mental health system are currently employed, although most clients with psychiatric disabilities (50%–70%) report wanting to work.11–17 The positive impact of sustained employment for individuals with SMI has been demonstrated.18–25 Employment provides opportunities for building social supports, daily activity and the structure essential to supporting individuals with SMI in their treatment and recovery. Benefits include increased self-esteem and independence, social integration, and community participation.4

    In this proof-of-concept pilot study, the investigators will examine the feasibility, acceptability and usefulness of the WorkingWell app as an innovative mobile employment support tool (ie, for use with a smartphone or tablet) for people with SMI managing in the workplace. The investigators will pilot WorkingWell to establish the logistical feasibility of implementing the app, as well as to evaluate the feasibility and usefulness of the research protocol. This pilot work will inform essential modifications to the WorkingWell app and refinements to the research protocol for the next steps in larger scale efficacy and effectiveness trials. The investigators will obtain preliminary data with standardised measures of social support, work confidence, app usability and job satisfaction in order to assess the usefulness of these measures and inform the power calculations and sample size for future studies.

    The individual placement and support model

    The individual placement and support (IPS) employment model has demonstrated robust success in promoting competitive employment among individuals with SMI.26 IPS promotes client choice and shared decision-making regarding employment plans, collaborative involvement of clients with the treatment team to identify and implement strategies to promote success and competence in finding competitive work, and ongoing support to help maintain a positive employment course.27 The effectiveness of IPS for individuals with SMI has been established in 23 randomised controlled trials.18 28–31 Approximately two-thirds of clients enrolled in IPS achieve the goal of a competitive job, compared with fewer than a quarter of clients who receive other forms of vocational services.32

    Despite the strong success of IPS in promoting competitive employment among those with SMI, a significant portion of individuals do not become steady workers, and relatively brief job tenures are common.19 32 Factors associated with job loss or retention include illness severity, duration and treatment; mental and physical well-being and functioning; neurocognitive capacities and social competence; work history33; the extent to which a job matches an individual’s interests, values and competencies34; motivation to work35; possession of skills to do the work and self-efficacy, for enacting work-related skills35 36; and ongoing support.34 37 38 The WorkingWell app, grounded in the underlying principles of IPS,27 is designed to extend the reach of employment supports, as well as address the limits of supported employment for individuals once they are on the job, for example, to promote social support, self-efficacy and motivation in the moment.

    The potential benefits of mobile technology

    Mobile communication devices (eg, smartphones) and cloud computing may be the most effective, least expensive and least stigmatising way to provide follow-up support to the broadest group of individuals with SMI in the workplace.39 40 Individuals with SMI demonstrate interest in and readiness to use technology-based tools.41–43 Of the respondents in a recent study of individuals with SMI, 72% reported that they own a mobile phone or smartphone and use it daily for a wide array of functions including texting, emailing and internet use.43

    Investigators have demonstrated that a smartphone intervention for illness management in individuals living with schizophrenia is feasible and acceptable, and may be clinically helpful in reducing symptoms.42 For example, FOCUS is a smartphone-based tool designed to help individuals with schizophrenia manage their symptoms.42 44 In initial trials, participants used the FOCUS system 86% of the days they had the device, and used on-demand resources beyond the built-in daily prompts. Level of cognitive functioning, negative symptoms and reading level did not hamper use of the system.42 44 This groundbreaking work clearly demonstrated that individuals living with SMI are interested, willing, and able to use and benefit from a smartphone intervention.

    Increasingly, individuals with SMI rely on internet-based and smartphone-based tools for health information and tracking.45–51 In a recent systematic review of 28 studies of e-mental health self-management interventions for individuals with psychotic disorders, researchers found that individuals were able and willing to use e-mental health services.52 Results also suggested that e-mental health services were at least as effective as usual care or non-technological approaches.52

    The WorkingWell mobile app intervention

    Coping on the job requires individuals with SMI to manage their symptoms while simultaneously dealing with the demands of the workplace and the expectations of their employer and coworkers. These challenges were identified in the previous discovery phase of this research.53 The target domains of the WorkingWell app include managing interpersonal relationships and social situations (eg, getting along with coworkers and supervisors); understanding job characteristics, tasks and expectations (eg, grasping the sequence of tasks and transitioning from one task to the next); dealing with illness-related and treatment-related issues (eg, coping with the side effects of medication); managing lifestyle/wellness and conditions apart from work (eg, sleeping or eating well, needing assistance with transportation or childcare); and sustaining motivation (eg, coping with the challenge of working in a job that is not interesting).53

    The WorkingWell app offers support through a variety of user experiences. WorkingWell users begin their engagement with the app by selecting up to three goals to work towards and on which to reflect in the coming week. Users are reminded of their goals the first time they open the app each day and have the opportunity to choose new goals each week. After choosing and reviewing their weekly goals, users are directed to the home page of WorkingWell, which features four main components: Manage the Moment, Remind Me, Rate My Day and My Progress (see figure 1, the WorkingWell home screen). Manage the Moment offers users a list of coping skills and tips for dealing with challenging situations and ideas on how to implement them. Remind Me provides users with a set of tools for setting text message reminders for themselves, creating to-do lists and taking notes. Rate My Day allows users to rate on a scale of 1 to 5 stars their effort in accomplishing their goals, and their success in a variety of other areas including dealing with stress and finishing tasks. My Progress displays a feedback message (eg, ‘Way to go! Things are going fantastic! What can you do to keep it up?’) based on users’ ratings, along with a detailed record of their entries for the past 4 weeks. Users also receive a motivational quote and image each day they use the app.

    Figure 1

    The WorkingWell home screen. This figure provides the image of the WorkingWell home screen that would be seen by the user after he or she pushes the WorkingWell icon on the smartphone screen. The four main components are clearly labelled, each with its own icon, to facilitate navigation to the component selected by the user.

    Methods and analysis

    Patient involvement

    The WorkingWell mobile app was developed by a team of researchers and providers in partnership with individuals with SMI; the Expert Advisory Panel, including distinguished supported employment researchers, intervention developers, trainers and providers, and individuals with SMI; and experienced app designers.53 WorkingWell addresses the needs identified by individuals with SMI in previous research, including strategies for addressing interpersonal relationships and social situations, job tasks and expectations, illness-related and treatment-related issues, lifestyle/wellness and conditions apart from work, and motivation.53 The app provides follow-up that does not require additional employment specialist or provider support.53 WorkingWell does not require disclosure of SMI or disability status in the workplace, consistent with recommendations in previous research and by the Expert Advisory Panel, and can reach those in the target population who do not have access to frequent face-to-face employment supports.53


    This pilot study will test the feasibility, acceptability and usefulness of the WorkingWell app, as well as the research protocol itself, and provide pilot data on potential outcomes. Mixed quantitative and qualitative methods will be employed in this 2-month pretest and post-test study. These include standardised measures, app usage metrics, and semistructured technical assistance and postimplementation exit interviews with study participants and agency staff. The investigators hypothesise that use of WorkingWell will be related to positive changes in social support, self-efficacy and motivation, and ultimately employment outcomes reflecting improved job tenure. Another purpose of the pilot study is to evaluate WorkingWell implementation: the feasibility of implementation with specific stakeholders (eg, those who are more experienced with smartphone technology vs those who are less experienced), user acceptance and satisfaction, and the likelihood of adoption as reflected in interviews with study participants and agency staff. In addition, the investigators will assess the feasibility and usefulness of the research protocol.


    Each participant will be provided with a Samsung Android smartphone with an unlimited data plan to access the WorkingWell app and communicate with research staff. These resources will be provided to all participants regardless of whether or not they already have their own smartphones to (1) eliminate having a personal smartphone as a participation criterion; (2) avoid creating a financial burden related to having to pay for their own data plans or potential overages on an existing plan; (3) ensure that participants have a reliable way of communicating remotely with research staff; (4) facilitate the research staff’s ability to provide accurate and standardised technical assistance; and (5) limit unnecessary variability among participants related to the phone’s user interface, operating system and others. Participants will only be able to access the WorkingWell app on the study-provided smartphone. Participants will be welcome to use the study phone for non-study-related purposes; user data not related to the use of the app (such as contacts, overall data use and others) will not be recorded or considered by the investigators.

    Partner agency sites

    The 40 participants who will enrol in the study and use the WorkingWell app will be recruited from six community mental health agencies located in the states of Maryland, Massachusetts and Vermont. This selection of agency sites represents a range of supported employment services, including IPS-supported and non-IPS-supported employment models. This will allow the investigators to compare the use of the app in the context of differing supported employment approaches. The selected sites have partnered with the investigators on research in the past, and agency leadership and staff are familiar with typical research requirements. The clients served by these agencies are individuals with SMI and related disabilities, making them eligible for public sector services and supports including supported employment. Participating agencies will receive a small stipend for their recruitment efforts and assistance.

    Sample selection

    The criteria for WorkingWell study recruitment will include the following: that the participants must be (1) 18 years of age or older; (2) receiving supported employment services (and, by definition of service eligibility, living with SMI); (3) working an average of 10 or more hours per week; (4) employed in a position that is not by definition seasonal or temporary; and (5) capable of reading and writing in English at a sixth-grade reading level or higher. Participants may be starting a new job or have been employed for any length of time at the point of study enrolment, as one pilot study objective is to learn whether the app is more or less useful to individuals at different stages of employment. Participants will not be required to have a minimum level of familiarity with smartphone or computer technology to enrol in the study; on the contrary, researchers will intentionally include participants with a wide range of pre-existing familiarity with this type of technology so that the usability of the WorkingWell app can be assessed across a diverse sample. Participants will receive a stipend of $25 for completing the orientation, $50 for completing the first half of the study marked by the completion of the fourth technical assistance call and $75 at the completion of their exit interview, for a total of $150.

    A sample size of 40 was deemed appropriate for this pilot study, given the exploratory nature of the study and the initial focus on feasibility, acceptability and usefulness, rather than statistical significance and intervention impact. Forty participants will allow researchers to see the ways people use the app and experience its value. This information will inform revision and tailoring of WorkingWell to make it more suitable for the majority of users. Finally, researchers will be able to take a preliminary look at the effectiveness of WorkingWell, both qualitatively and quantitatively, to guide future research.


    A liaison will be designated at each agency site. Their responsibilities will include assisting in the recruitment and screening of potential participants, facilitating study visits and communication, and providing on-site technical assistance as needed. Prior to the start of the recruitment and screening process at each agency, research staff will meet with agency leadership and staff to describe the study and the WorkingWell app, and explain participant eligibility criteria and the role of the agency liaison. Interested agency staff and leadership will also be given preview access to WorkingWell on their personal smartphones for their own use and as a demonstration tool to be used in recruiting participants.

    Designated agency liaisons will be provided with recruitment materials to distribute to potential participants (including an information sheet about the study and a consumer task list describing the tasks that participation will entail). Liaisons will also be provided with an eligibility screener checklist of selection criteria to complete for each potential participant. When a potential participant is identified, the agency liaison will contact the research staff to review the eligibility screener on the telephone or in an inperson meeting at the agency to confirm the participant’s eligibility.

    Eligible individuals will then be invited by the agency liaison to enrol in the study, beginning with attendance at an orientation session. The agency liaison will coordinate this scheduling with the study team and potential participants to determine mutually agreeable dates and times, as well as to reserve a comfortable meeting space at the agency site.

    Study enrolment and participant orientation

    Research staff will travel to agency sites to enrol participants and provide the inperson orientation, allowing approximately 90 min for these initial sessions. These may be either individual or group sessions. Research staff will first obtain written informed consent from the attendees. Participants will be provided with a copy of the informed consent document. The consent process will include an explanation of study tasks and how data will be collected, and an overview of the interviews and surveys participants will be asked to complete. Research staff will also engage participants in a discussion of appropriate smartphone use in the workplace (eg, using the WorkingWell app during a lunch break or before or after work rather than while on the job) to discourage participants from using the smartphone in a way that may negatively impact their employment or safety. As part of the informed consent process, research staff will discuss the risk of loss of privacy associated with participation in a research study and how those risks will be mitigated. The WorkingWell app is not meant to elicit sensitive or identifying information from users. However, because users will have the option of entering their own goals, notes and reminders in the app, they have the potential to insert such information themselves. All user data, including that which may be sensitive or identifying, are stored in a password-protected database on a secure remote server. Should researchers be led to believe, either through user data or direct communications, that a participant is at risk of harming themselves or others, they will follow standard safety protocols, as described in detail in the consent form.

    Agency liaisons will be present at these initial orientation sessions for informational purposes and will be encouraged to attend subsequent orientation sessions at their agencies when possible. Employment specialists who are interested in the WorkingWell app or whose clients are participants in the study will also be encouraged to attend the orientation.

    Participants will be assigned unique study identification numbers. They will be asked to complete the paper-based presurvey. After collecting the participants’ presurveys, research staff will distribute study smartphones (with unlimited data plans) and study refrigerator magnets (for recording app usernames and passwords) to participants. All participants, regardless of whether or not they already have their own personal smartphone, will be provided with a Samsung smartphone that has an Android operating system. If there are changes in equipment availability, different but similar models of Samsung smartphones will be used over the course of the study. Efforts will be made to distribute the same smartphone model to participants within agency sites to minimise technical assistance confusion or inconsistencies that could arise from different models being used within the same agency. Study phones will be preprogrammed with the WorkingWell icon on the home screen, a screen lock code (which a participant may change) for security reasons and research staff contact information. Participant login information will be recorded by research staff in an encrypted spreadsheet for reference in the event that participants forget or lose their passwords. Research staff will verify that participants are able to unlock the phone and log into the app, and provide one-on-one assistance as needed.

    Research staff will review a study phone user guide and WorkingWell User Guide54 while participants follow along in performing phone and app functions on their study phones. These guides have been created by research staff as supplements to hands-on training (see online supplementary file: WorkingWell User Guide). User guides will be displayed on a projector for larger group orientations (more than five participants). Research staff will review the consumer task list with participants, address any questions and schedule their first technical assistance calls for the following day.

    Supplementary file 1

    Technical assistance

    Research staff will conduct five technical assistance telephone calls individually with each participant over the course of the 2-month pilot study by calling participants on their study phones or personal phones depending on the participants’ expressed preference. The first technical assistance call will be conducted 1 day after orientation to ensure that no technical issues with the study phone or application have arisen. The subsequent technical assistance telephone calls will be conducted at 1, 2, 4 and 6 weeks postenrolment in the study. If research staff are unable to contact a participant for a technical assistance telephone call within a given period of time, the agency liaison for that participant’s enrolment site will be notified to help facilitate communication. This agency-based support is designed to help minimise participant attrition and encourage engagement with these calls. The purpose of the technical assistance calls is to monitor any challenges participants may encounter with the app or the phone, to provide technical assistance as needed, to understand participants’ developing use and impressions of the app, and to address any questions or concerns that the participants may have in regard to the study or the app. In the case that a participant misses two consecutive technical assistance calls or is out of communication with research staff for more than 3 weeks, they will be considered lost to follow-up. Participants will also be encouraged to call or text research staff outside of their scheduled technical assistance calls if they need additional support or have questions or feedback. Participants may also receive technical assistance from agency staff, friends and family members. All communications between participants and research staff will be recorded for implementation assessment purposes. Research staff will also provide agency staff with a log for recording any technical assistance they facilitate.

    It should be noted that the technical assistance telephone calls may have the unintended effect of prompting or increasing participants’ engagement with the app. However, the investigators have determined that these periodic communications will be integral for understanding the feasibility and usability of the app, and that addressing participants’ technical or other challenges will be necessary to ensure that individuals with varying levels of technological experience are adequately supported.

    Exit sessions

    At the conclusion of each individual’s participation, research staff will return to their agency site for an exit session. Participants will be asked to complete a paper-based postsurvey featuring repeated measures from the presurvey plus additional items assessing impressions of the app and their participation in the study. Research staff will conduct inperson exit interviews with participants. The interviews will assess user experience related to characteristics of the app (eg, general impressions, ease of use, design quality, need for technical assistance, any problems that may arise with the app) as well as users’ impressions of the research experience (eg, orientation session training). The investigators will delete any saved data via a factory data reset from each participant’s study phone before collecting the phone.

    Agency liaisons or employment specialists whose clients participate in the study will be invited to complete exit interviews with research staff. These interviews will focus on staff impressions of the feasibility, acceptability and usefulness of the WorkingWell app based on their own use and their observations of their clients’ use of the app.

    Outcomes and measures

    Background and demographic data and data pertaining to participants’ prior experiences with and use of smartphone technology will be obtained in structured items in the presurvey administered to participants in the orientation session. Data on job tenure will be obtained via participant report in the postsurvey administered prior to the exit interview, including employment status, number of different jobs worked over the course of the study, number of weeks employed during study, changes in employment (if any) and time between jobs, and average number of hours worked per week (both at the beginning and end of the study).

    App usage tracking tools and usage analytics will be used to obtain indicators of the feasibility and acceptability of WorkingWell. Participant app data will be downloaded and monitored on a daily basis for quality assurance and app use tracking purposes. Data will include participants’ daily number of navigations to the home screen of the app, Rate My Day check-in reports (eg, check-in ratings, reported skill use), Manage the Moment navigations, participants’ selected weekly goals, and participant-generated notes, task lists and text message reminders in Remind Me.

    Feasibility, acceptability and usefulness will also be assessed using data obtained in technical assistance calls with study participants and exit interviews with participants and agency staff. The technical assistance calls and exit interviews will consist of a structured script and questions, and include items soliciting information on general impressions of app use, ease of use, design quality and specific needs for technical assistance, as well as impressions of the research experience (eg, orientation session training, WorkingWell User Guide, methods and measures). A subset of questions will be repeated in each technical assistance call and in the exit interview to monitor changes in participants’ responses over the course of their participation. Additional questions will be added to later technical assistance calls to elicit increasingly detailed feedback as participants’ experience with the app increases.

    Participant motivation will be assessed using the six-item interest/enjoyment subscale of the Job Match Survey (JMS),55 56 rated on a 7-point Likert-type scale (‘not true at all’ to ‘very true’). Participants’ responses to these items reflect their connection to work (eg, ‘I enjoy doing this job very much’, ‘My job is fun to do’, ‘I think this job is boring’). This subscale of the JMS has been found to have good reliability and validity in prior studies of similar individuals.34 In a recent study of individuals with SMI receiving supported employment services, the complete JMS (Cronbach’s alpha=0.90) and, in particular, the interest/enjoyment subscale (0.86), were found to have good internal consistency. This construct was most highly related to job tenure outcomes for individuals in IPS, which underscores its relevance to the current project.34 57

    Self-efficacy will be assessed by a subscale of the Work Related Self-Efficacy Scale (WRSES): General Work Skills Self-Efficacy (12 items).36 The complete 37-item WRSES was developed in a 12-month longitudinal survey of urban residents diagnosed with serious psychiatric disability (schizophrenia or schizoaffective disorder). All items have demonstrated good test–retest reliability and good concurrent validity with employment variables.36 The General Work Skills subscale has demonstrated good psychometric properties in studies of employment of individuals with psychiatric disabilities, with high internal consistency (0.94), and is associated with employment outcomes.36

    Social support will be assessed as perceived by participants using the 18-item Medical Outcomes Study Social Support Survey (MOS-SSS).58 59 This brief, multidimensional, self-administered survey addresses four functional support dimensions (emotional/informational, tangible, affectionate and positive social interaction) and provides for the construction of an overall functional social support index. The instrument has demonstrated reliability and validity in studies with individuals with psychiatric disabilities. In this pilot study, researchers will use an eight-item subscale addressing the emotional/informational dimension of the MOS-SSS. The full-length version has a Cronbach’s alpha of 0.88, and in field testing showed good convergent and discriminant validity.58 It has been validated on US and Canadian patients with chronic conditions, and has been used successfully in the investigators’ previous research on mothers with psychiatric disabilities.60

    A subset of seven items of the System Usability Scale (SUS) will be adapted to reflect app use, and used to measure participants’ perceived usability of the WorkingWell app.61 62 Participant responses are rated on a 5-point Likert scale (‘strongly disagree’ to ‘strongly agree’) to assess app complexity, ease of use, confidence in app use, and need for assistance or prior experience to successfully use the app. The SUS has been demonstrated to have good validity and has been used on small sample sizes with reliable results.61 62

    Data analysis plan

    Data will be collected, managed and analysed throughout the project. All surveys will be visually inspected for completeness on collection to minimise missing data. Survey data and categorical interview data will be entered into databases using Qualtrics survey software.63 Data will be analysed using standard statistical software to describe participants (eg, SAS V.9.2 or SPSS V.16.0).64 65 Quantitative data obtained in response to structured survey items will be entered, cleaned, analysed and summarised. Observational and interview data, including technical assistance calls and exit interviews, that are obtained in response to open-ended items will be analysed qualitatively for content and themes that emerge, and coded and categorised using Dedoose software.66 Results from qualitative analyses will be cross-tabulated in matrices with aggregated quantitative data to explore relationships among participants’ characteristics and previous experiences, users’ interactions with the app, and subjective reports of feasibility, acceptability and usefulness of the app.67 The technical assistance call data from participants who drop out of the study or are lost to follow-up will be included in the qualitative analysis.

    The investigators will use descriptive statistics to describe the study group and to summarise their use of various components of WorkingWell over the course of the 8 weeks of study participation. The relationships between their navigations to the home screen, the Manage the Moment screen, the Remind Me screen, the My Progress screen and the Rate My Day screen, and the number of completed check-ins in Rate My Day and responses to presurvey and postsurvey items, such as ratings of the ease of phone use and SUS app usability scores, will be assessed. Participant-entered text will be analysed qualitatively to assess trends in the ways in which participants use Remind Me and the types of personalised goals they create. These analyses will address research questions regarding phone and app use, and the suitability, necessity and sufficiency of the orientation and technical assistance.

    Paired t-tests will be used to examine pretest versus post-test changes in participants’ reports of social support, self-efficacy and motivation. In addition, researchers will use one-sample t-tests to compare study post-test outcomes against agency employment rates and averages. These comparisons will provide a context within which to judge the relative effectiveness of WorkingWell and will inform subsequent randomised controlled trials. For those participants who are determined by researchers to be lost to follow-up, pretest data will be used for descriptive purposes; however, their data will be eliminated from pretest-post-test comparisons.

    The investigators will assess participants’ engagement with WorkingWell and what strategies and resources need to be provided to support the initiation of use of the app across a range of users. The investigators will assess how participants use the app and what patterns of use develop over time. Participants will also be asked about the helpfulness of the app in supporting their work lives, both for starting a new job and for sustaining a job.

    Strengths and limitations

    This pilot study will provide preliminary data on the feasibility, acceptability, usefulness and impact of the WorkingWell app, and provide feedback on the feasibility and usefulness of study measures and procedures. A mixed-methods approach, employing standardised measures as well as semistructured interviews with participants and agency staff, will provide rich data on the impact and experience of using the app, along with recommendations for improvements in the app and research protocol. Given the exploratory, proof-of-concept nature of the study, the sample size will be small, and a pre-post design is deemed most appropriate; there will be no comparison or control group in the pilot study design. A larger sample size would allow for stratification of the sample by individual characteristics potentially related to mobile app use and effectiveness, such as specific psychiatric diagnosis, cognitive impairments, or related physical or sensorimotor functioning. A larger sample size or targeted sample would allow for exploration of the feasibility and usefulness of WorkingWell in diverse employment contexts.

    Ethics and dissemination

    Study findings will be disseminated to agency and service recipient partners, state agencies and funders, and to the research and technology development communities. Papers will be prepared for presentation at professional meetings and publication in scientific journals. Study materials (eg, data collection and interview protocols, WorkingWell User Guide) will be available for use by research colleagues in future studies.

    Participants’ challenges with and suggestions for the WorkingWell app (eg, app layout and content), and the research protocol and procedures (eg, orientation to the app, technical assistance and data collection tools) will be used to inform future versions of the WorkingWell app and future studies with potentially diverse samples in a range of settings. For example, if pilot study findings suggest that participants with certain sets of personal or job characteristics are more or less able or willing to use and benefit from WorkingWell, modifications to the app may be recommended. Positive trends for particular subsamples of participants, although small in number in this pilot, may suggest the potential benefit of testing WorkingWell with targeted groups of users or with samples large enough to allow for stratification by individual or workplace characteristics. Pilot feasibility study findings will directly inform the next steps in app development, adaptation and testing.

    Success at work is important to individuals living with SMI for many reasons. Employment provides purpose and meaning in life, a sense of ‘normalization’, a vehicle for community inclusion, and essential resources to individuals and families who are likely to be marginalised by the impairments and stigma conveyed by SMI and the common socioeconomic correlates of limited resources and poverty.68 Failure in this important life role may undermine an individual’s treatment engagement and recovery, and the resilience of future generations of family members. An accessible, usable and useful tool that supports optimal functioning, well-being and success on the job would be a powerful adjunct to treatment and a potentially profound contributor to recovery. In this pilot feasibility study, the investigators will take the initial steps towards demonstrating the effectiveness of a tailored, individualised technology-based tool for individuals with SMI. This work will add to the field of supported employment research in general, and has specific implications for the development and testing of tools to support the success of individuals with SMI in other important life domains.

    Study status

    Screening and recruitment for the study commenced in February 2017. The study is ongoing until July 2018.


    We thank the agency sites and app users with serious mental illness for their ongoing contributions to the study. T Chris Burns, MFA, was instrumental in WorkingWell app development. Sarah E Lord, PhD, Elizabeth Carpenter-Song, PhD, Justin S Tauscher, MS, and Lynn H MacPherson, BA, contributed to earlier phases of this work. Dror Ben-Zeev, PhD, Rachel Brian, MPH, and Geneva Kay Jonathan, BA, provided consultation on the app testing process. Gregory J McHugo, PhD, and Mary Ann Greene, MS, serve as research consultants to data management, manipulation and analysis.


    1. 1.
    2. 2.
    3. 3.
    4. 4.
    5. 5.
    6. 6.
    7. 7.
    8. 8.
    9. 9.
    10. 10.
    11. 11.
    12. 12.
    13. 13.
    14. 14.
    15. 15.
    16. 16.
    17. 17.
    18. 18.
    19. 19.
    20. 20.
    21. 21.
    22. 22.
    23. 23.
    24. 24.
    25. 25.
    26. 26.
    27. 27.
    28. 28.
    29. 29.
    30. 30.
    31. 31.
    32. 32.
    33. 33.
    34. 34.
    35. 35.
    36. 36.
    37. 37.
    38. 38.
    39. 39.
    40. 40.
    41. 41.
    42. 42.
    43. 43.
    44. 44.
    45. 45.
    46. 46.
    47. 47.
    48. 48.
    49. 49.
    50. 50.
    51. 51.
    52. 52.
    53. 53.
    54. 54.
    55. 55.
    56. 56.
    57. 57.
    58. 58.
    59. 59.
    60. 60.
    61. 61.
    62. 62.
    63. 63.
    64. 64.
    65. 65.
    66. 66.
    67. 67.
    68. 68.


    • Contributors JN is the principal investigator and is responsible for the design and implementation of the study. JN, SMW and AMC codeveloped the research protocol, procedures and necessary modifications, and prepared and submitted relevant materials for ethics approval. SMW and AMC are implementing the protocol in community-based settings, with oversight and review by JN. JN, SMW and AMC wrote the manuscript together, each drafting sections, and reviewing and editing all components of the manuscript. All authors read and approved the final manuscript.

    • Funding This work is supported by the US National Institute on Disability, Independent Living, and Rehabilitation Research grant #90IF0069, WorkingWell: Developing a Mobile Employment Support Tool for Individuals with Psychiatric Disabilities. The views expressed in the submitted article are those of the authors and not the official position of the funder.

    • Competing interests None declared.

    • Patient consent Not required.

    • Ethics approval Ethics review and approval were obtained from the Dartmouth-Hitchcock Medical Center Committee for the Protection of Human Subjects (#00028834), the Massachusetts Department of Mental Health Central Office Research Review Committee (#2015-21) and the Vermont Agency of Human Services Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Collaborators Gregory J McHugo and Mary Ann Greene.