Phase 1: Development of items, text and images for ADOC-E

We recruited 14 experienced occupational therapists from four countries to participate in phase 1 of the study (four from USA, two from UK, four from Australia and four from New Zealand). The participating occupational therapists had an average of 13.3 years (SD 5.2) clinical experience. The majority (79%; 11/14) had postgraduate clinical qualifications, with two holding a doctoral degree and three further participants being doctoral candidates, with the remaining three holding bachelor degrees as their highest qualification. The occupational therapists worked in acute medical wards, specialist neurological units, mental health services, independent living facilities, universities and community rehabilitation services.

All 14 participants contributed to the first survey round. However, two participants (one from the USA and one from New Zealand) dropped out of the study after the first survey round due to other work commitments. One additional participant from the USA missed the second survey round but contributed to all three other survey rounds.

We began with 97 ADOC-E items derived from the original Japanese iPad application after removing four specially Japanese activities (eg, budo—a Japanese martial art; gētobōru—a Japanese game like croquet). These 97 ADOC-E items were divided into eight groups based on groupings in the Japanese version of ADOC: self-care (6 items); mobility (11 items); domestic life (15 items); work and learning (4 items); interpersonal interactions (5 items); social activities (7 items); sport and physical activities (20 items) and hobbies (29 items). In the first Delphi round, 98% (95/97) of these items were considered relevant and meaningful as potential goals in rehabilitation, although suggestions to improve the scope of items or the language used to describe them were provided for 68% (66/97) of the ADOC-E items. The two items which did not meet our 80% agreement for inclusion at this stage were for ‘baseball’ (71% agreement) and ‘softball’ (79% agreement), with a few participants indicating that these were not common sports in their countries. Indeed, the whole area of ‘sport and physical activity’ was particularly difficult to reach consensus around, as different sports are important in different countries. Furthermore, the language used to describe sports differs from country to country, for example, the meaning of the word ‘football’ is highly contested. To address this problem, we decided to collapse all ‘like’ sports into groups represented by a single ADOC-E item, for example, ‘team ball sports’, ‘racket and ball sports’ and so on.

Another area of contention was the use of UK versus US spelling in ADOC-E. To address this problem, we revised ADOC-E to avoid these words altogether, for example, replacing the term ‘radio programme’ (UK spelling)/‘radio program’ (US spelling) with ‘radio show’. Where participants from different countries disagreed on the correct term to describe certain objects or activities, we revised ADOC-E to include both sets of language, for example, referring to ‘getting petrol/gas’ in the ADOC-E item related to ‘vehicle maintenance’.

In terms of rephrasing the text, the participants’ recommendations focused primarily on clarifying the scope of each ADOC-item and on replacing clinical or technical terms with more colloquial language, for example, the ADOC category for ‘mobility’ was retitled ‘getting around’. Following recommendations in this first Delphi round, we also added several new ADOC-E items. For instance, ‘sleeping’, ‘relaxation’ and ‘dealing with emergencies’ were added to the ‘self-care’ items. Going into phase 2 of the Delphi process, we had revised ADOC-E to include 98 items but with a few more in ‘self-care’ (9 items) and ‘work and learning’ (5 items) and a few less in ‘sport and physical activity’ (17 items) and ‘hobbies’ (28 items).

In the second Delphi round, 100% of the items were considered relevant and meaningful as potential goals in rehabilitation by all participants but with a few more minor suggestions to improve the text for 33% (33/98) of the items. Some recommendations at this stage were to include terminology that was only relevant in one country (eg, referring to a specific social service by name that was country specific). As we intended ADOC-E to be internationally accessible, we avoided these types of changes.

‘Gambling’ as a hobby, which was included in the original Japanese ADOC application, was a particularly controversial ADOC-E item that provoked a lot of debate. Some questioned whether it should be included because it could be considered a vice and detrimental to the health and well-being of certain individuals. In the end, however, there was overall approval to include this item in ADOC-E for a number of reasons. Including this item reflected an acknowledgement that some people enjoyed gambling as a recreation and had a right to make their own personal choices about such activities. Participants also reflected that it was better for health professionals to know and talk about this aspect of clients’ lives if it was an issue for them than for it to be a hidden problem.

Following recommendations in this second Delphi round, we added four new ADOC-E items: ‘using hearing aids, glasses, contact lenses’ under the ‘self care’ items; ‘water sports—sailing, windsurfing, surfing, rowing and so on’ and ‘mountain sports—skiing, snowboarding, rock climbing, mountain climbing and so on’ in ‘sport and physical activities’ and ‘attending educational talks or seminars for enjoyment’ under ‘hobbies’. We also removed two ADOC-E items that were, after revision of the text, very similar in content to other ADOC items. This left us with 100 ADOC-E items for round three of the Delphi process.

In the third Delphi round, we presented the revised ADOC-E items with both their text and images. Of the 100 ADOC-E images, 91 reached our criteria for agreement by the participants, that is, 80% (>9/12) agreed the images were appropriate and acceptable, with 62 images being accepted by all 12 participants. Suggestions for further improvements were provided for 22 images (including the nine that did not reach our threshold of ‘agreement’ by the group). We revised these 22 images as recommended, and in our fourth Delphi round, canvassed the participants for their opinions on these revisions. In the fourth Delphi round, we reached our threshold for agreement, with over 80% (>9/12) of the participants indicating the revised ADOC-E images and text were appropriate and acceptable. All 100 agreed images were included in the next phase of our study. Raw data from each Delphi round is provided in online supplementary files 1–4. The text from the original ADOC-E translated from Japanese and the final 100 agreed ADOC-E items are provided in online supplementary file 5.

Phase 2: Testing the suitability of ADOC-E images with health service users

Twenty-four health service users (10 male; 14 female) from three disability and rehabilitation organisations took part in phase 2 of this study. Their mean MMSE score was 25, with a range of 13–30, indicative of severe to no cognitive impairment.31 All participants were above the MMSE score cut-off for using ADOC (see table 1).32 The median number of ADOC-E items that each participant commented on was 52 items (IQR: 66.25 items; range: 11–100 items). Eight participants commented on all 100 ADOC-E items. Each individual ADOC-E item was considered by 13–16 different people.

Cohen’s kappa, as a measure of agreement between the two researchers categorising the participants’ open recognition of the ADOC-E items, was 0.77, indicative of substantial agreement. Where the two researchers disagreed, the field researcher tended to have been more conservative in his judgement regarding open recognition (ie, more likely to consider a response non-recognition than the second researcher). In our reporting of participants’ responses below, we have retained the more conservative scoring of image recognition.

Overall, 40 of the 100 ADOC-E images were correctly identified by the participants without prompting (ie, open recognition) all of the time. Correct recognition of 72 of the 100 ADOC-E images was achieved by all participants when prompted recognition was also an option. Ninety-five of the 100 ADOC-E images were correctly identified by 80% of participants when both unprompted and prompted recognition was counted. Five ADOC-E images could not be correctly identified even with prompting for 20%–40% (3-6/15) of participants. The images which proved most difficult for participants to identify tended to be those related to social participation. Specifically, these five difficult ADOC-E items were for ‘paid employment’—represented by a person working on a computer in an office; ‘voluntary (unpaid) work’—represented by a person picking up rubbish in a park; ‘higher education/vocational training’—represented by a picture of a university student studying books; ‘public interactions’—represented by a person getting direction from a stranger on the street and ‘using public services’—represented by a person getting a book out of a library. These difficult ADOC-E items represented more abstract concepts and more individualised social roles than more easily recognisable ones such as, for instance, images representing ‘using the toilet’, ‘shopping’ or ‘playing golf’.

For 19 of the 100 ADOC-E images, over 80% of the participants thought the picture did not accurately represent the intended concept. Frequently, in these cases, the participants focused on the specific nature of the picture rather than the broader concept we wanted it to represent. Again this was more of a problem for the more abstract concepts. So, for instance, while 100% of participants (14/14) correctly identified a picture of people going to the cinema, 50% of people did not consider this a good representation of the broader ADOC item ‘enjoying arts—going to shows/concert/art galleries/movies/museum’ because it was just a representation of one of these activities. In fact, 9 of these 19 images were considered problematic by participants even though they were correctly identified, to the satisfaction of the researchers, by 100% of the participants without prompting. The participants also provided useful feedback on confusing or distracting aspects of images—such as pictures of people in cars who were not wearing seatbelts—that could be easily corrected. See online supplementary file 6 for the raw data from this phase of the study.