Mixed methods study to develop the content validity and the conceptual framework of the electronic patient-reported outcome measure for vascular conditions

Objective The aim of this paper is to describe the stages undertaken to generate the items and conceptual framework of a new electronic personal assessment questionnaire for vascular conditions. Design A mixed methods study: First a survey of vascular clinicians was completed to identify the most common conditions treated in vascular clinics and wards. Quantitative systematic reviews were done to identify validated patient-reported outcome measures (PROMs) for direct inclsuion in the new instrument. However, due to scarcity of validated PROMs, the items of the new instrument were mainly based on a large qualitative study of patients and systematic reviews of the qualitative evidence. This was followed by a quantitative clinicians’ consensus study and, finally, a qualitative face validity study with patients. Participants Vascular patients participated in the primary qualitative study and the face validity study. In the qualitative study, 55 patients were interviewed, and for the face validity, 19 patients gave feedback. Twelve clinicians completed the survey and 13 completed two cycles of the clinicians’ consensus study. Results The items and scales in the electronic personal assessment questionnaire for vascular conditions (ePAQ-VAS) were generated based on the results of five systematic reviews evaluating existing PROMs for possible inclusion in ePAQ-VAS, five systematic reviews of qualitative evidence, a primary qualitative study involving 55 patients and clinicians’ input. One hundred and sixty-eight items were initially generated, of which 59 were eliminated by the expert panel due to repetition. The instrument was divided into one generic and three disease-specific sections (abdominal aortic aneurysm, carotid artery disease and lower limb vascular conditions). In each section, items were grouped together into putative scales. Fifty-five items were grouped across eight scales; the remaining items were kept as individual items, because of relevance to service users. Conclusions This multidimensional electronic questionnaire covers the most common vascular conditions. This is particularly important for patients presenting with mixed symptoms or multiple conditions. This tool captures symptomatology, health related quality of life (HRQoL) and other clinically relevant data, such as experience with services and comorbidities.

I, the Submitting Author has the right to grant and does grant on behalf of all authors of the Work (as defined in the below author licence), an exclusive licence and/or a non-exclusive licence for contributions from authors who are: i) UK Crown employees; ii) where BMJ has agreed a CC-BY licence shall apply, and/or iii) in accordance with the terms applicable for US Federal Government officers or employees acting as part of their official duties; on a worldwide, perpetual, irrevocable, royalty-free basis to BMJ Publishing Group Ltd ("BMJ") its licensees and where the relevant Journal is co-owned by BMJ to the co-owners of the Journal, to publish the Work in this journal and any other BMJ products and to exploit all rights, as set out in our licence.
The Submitting Author accepts and understands that any supply made under these terms is made by BMJ to the Submitting Author unless you are acting as an employee on behalf of your employer or a postgraduate student of an affiliated institution which is paying any applicable article publishing charge ("APC") for Open Access articles. Where the Submitting Author wishes to make the Work available on an Open Access basis (and intends to pay the relevant APC), the terms of reuse of such Open Access shall be governed by a Creative Commons licence -details of these licences and which Creative Commons licence will apply to this Work are set out in our licence referred to above.
Other than as permitted in any relevant BMJ Author's Self Archiving Policies, I confirm this Work has not been accepted for publication elsewhere, is not being considered for publication elsewhere and does not duplicate material already published. I confirm all authors consent to publication of this Work and authorise the granting of this licence.

Strengths and limitations of the study
 This electronic patient assessment questionnaire for vascular conditions was developed with input from patients and clinicians. We used mixed-methods to ensure content and face validity.  We examined the validity of existing patient reported outcome measures for vascular conditions and identified a lack of a holistic electronic measure. We systematically reviewed the qualitative evidence and recruited patients to generate qualitative data. The themes generated were used to develop most items.  Vascular clinicians were surveyed to ensure clinically relevant conditions and questions were included. They were also consulted in a consensus study.  This holistic assessment tool can be incorporated to the electronic patient records. This can help provide patient focused care, as well as be used to assess outcomes of vascular patients' overtime.  The burden of questionnaire is its main limitation; however, by providing strict skipping rules only relevant questions are presented to patients. This tool will be examined for other psychometric validity and reliability in future studies. Many generic and condition specific PROM have been adopted to examine impact of vascular conditions on patients and measure outcomes. This is despite a lack of evidence that they have been developed and evaluated in-line with accepted guidelines; in addition, these instruments are rarely used or formally evaluated in routine patient assessment in day to day clinical practice. We conducted scoping searches and informal discussions with vascular clinicians to identify any existing PROM, however, these preliminary stages in the research process suggested an absence of valid and reliable PROM for use in vascular populations.
In this paper, we report the stages in developing an electronic personal assessment questionnaire for vascular patients (ePAQ-VAS). This includes: (iii) Developing the items within each section of ePAQ-VAS based on qualitative systematic reviews (9-12), and a primary qualitative study (13). (iv) A consensus study with clinicians to rate the relevance of included items, and to add items to ePAQ-VAS based on the opinion of vascular surgeons, radiologists and nurses.
(v) A face validity study with vascular patients to examine the clarity and relevance of the items within ePAQ-VAS.
The aim of these steps was to develop a single electronic instrument covering most vascular conditions in line with international guidance (3). The conceptual framework and items were developed in a way to ensure this assessment tool can be used in patients with mixed symptoms and multiple vascular conditions.

Methods
Clinicians involved in the care of vascular patients were invited to identify the common vascular conditions treated by vascular surgeons and vascular specialists.
They were asked to list the key issues, symptoms and the impact of these conditions on patients suffering with these diseases. Data from this round were used to inform qualitative evidence synthesis.

Analysis of the qualitative evidence
Qualitative data from the primary study and each of the systematic qualitative reviews were analysed separately. Framework analysis was used to analyse the interviews (15). This analysis includes five stages, the first involved familiarization by reading of the transcripts and reading the primary data.
-The second stage involved identification of a thematic framework; the thematic framework was based either on clinical opinion for areas with no valid PROM, such as AAA and CAD, or a combination of clinical opinion and, when available, the scales of PROM with good content validity.
-In the third stage the data was coded and indexed by applying the thematic framework to the whole data set.
-At the fourth stage, a framework matrix was created by arranging the data per the thematic references.
finally mapping and interpretation, including examining patterns within the data and associations with it.

Clinicians input and consensus exercise
Twenty-three clinicians involved in the care and management of patients with vascular conditions were invited to a survey to list the most common vascular conditions managed by them and to list the key issues, symptoms and the impact of these diseases on patients. Data from this round were used to inform qualitative evidence synthesis.
Different group of clinicians involved in the care of vascular patients were invited to a consensus study to score the relevance of items (questions) in the provisional version of ePAQ-VAS. In total, thirty clinicians including vascular surgeons, interventional radiologists, vascular nurses, physiotherapists and occupational therapists were invited. Participants were asked to rate the appropriateness of each question on a five- point Likert scale of "Strongly disagree" (=0) to 'Strongly agree' (=4). This process was repeated, and members of the clinicians' panel were presented with the aggregate findings of the previous round, and again asked to score each question. This process aimed to examine the relevance of each item from the clinicians' perspective and to identify any new items suggested by the clinicians (16).

Developing scales and items
The ePAQ development team (AA, EL, PP, GJ, SR) employed an iterative process, incorporating evidence from the systematic reviews, qualitative study, and the clinicians' consensus study. In line with the FDA guidance (3), items (questions) were developed from the qualitative data using the following three steps: interpretation, translation, and triangulation of themes.
Interpretation involved familiarisation with the language used in the primary data included in the synthesis. This enabled translation of descriptions of apparently diverse issues affecting vascular patients into a single set of harmonised themes. The resulting themes were used to develop the items for ePAQ-VAS. The items were grouped into sections, and each section, further divided into scales consisting of a connected group of items. Triangulation was performed across evidence sources to ensure the items comprehensively covered all issues of importance to patients with AAA, PAD, CAD, VLU, and VV.

Face validity of ePAQ-VAS
A second phase of semi-structured patient interviews was conducted by (EL, PP) with 19 participants, purposeful sampled from the vascular populations previously described. ePAQ-VAS (version 1) was presented to participants and a focused interview was conducted to investigate vascular patients' perceptions of the questionnaire as a whole, and individual items. Questions were asked under the following headings of: -overall impressions, Interviews were audio taped, transcribed and analysed. A pragmatic approach was used for the analysis, with comments collated and presented back to the working group who made consensus decisions on revisions to ePAQ-VAS.

Results
In total twelve clinicians completed the first survey and identified PAD, AAA, VLU, VV and CAD as the most common vascular conditions treated by them. They listed common issues such as pain on walking, rest pain, reduced mobility or lack of mobility for patients with PAD, no physical symptoms for those with AAA but need for multidisciplinary approach to manage these patients. For patients with VLU, the main issues included burning pain, recurrence and healing; for patients suffering with VV, skin changes, appearance of leg and ulcer as well as ache were the main issues raised. The clinicians felt the key issue for patients with CAD was identifying patients benefiting from intervention and reducing the risk of stroke. The result from this survey was used to inform the analysis of qualitative data used to develop ePAQ-

VAS.
Systematic reviews and assessment of psychometric evaluation were conducted for PROM validated for use in PAD, AAA, VLU, VV and CAD. A total of 33 PROM that had undergone some form of validation were identified in 41 studies (Table 1.). conditionspecific

Condition Number of Citations
The most valid and reliable condition specific PROM was VLU-QOL.
Abbreviations: Abdominal aortic aneurysm (AAA), Carotid artery disease (CAD), Peripheral arterial disease (PAD), Venous leg ulcer (VLU), Varicose veins (VV). Table 1: Results from the systematic reviews of psychometric evaluation of vascular PROMs No PROM were identified that had undergone sufficiently rigorous development and validation to suggest that they were suitable for direct use in ePAQ-VAS, the details of these reviews have been reported previously (4-8). Where evidence existed, this fell short of required standards (3,14). For instance, the review investigating VV PROM (4) found some evidence for, and discussion of, content validity in relation to the Aberdeen varicose vein questionnaire (AVVQ) and suggest that it is the most appropriate existing condition specific measure for use in a VV population. However, item generation for this PROM involved a literature review and assessment by clinicians of relevance of included items with no direct involvement of patients, therefore suggesting a deficiency in terms of content validity (17). The scales from these reviews were used to provide a framework for the systematic qualitative reviews and the primary qualitative study (5, 9-13).  Table 2.

Condition
Sample size Key Findings AAA 13 No physical symptoms, a small number of participants reported abdominal pain and pain in their legs. Uncertainty, anxiety and fear of rupture and death appeared to impact most greatly on people's QoL. CAD 9 This condition seemed to have had the least impact on physical and social function, although psychologically it created a sense of worry and anxiety for some participants. The main reported outcome was fear of having a major stroke. PAD 14 Pain and mobility were the most commonly reported themes. The extent to which they impacted on QoL was associated with the severity, age expectations and social support. Fear of the symptoms worsening and amputation was evident. VV 10 VV do not appear to have had a major impact on overall QoL for most the participants. Pain was the most common issue. The perceived unpleasant appearance of the VV seemed to have the greatest psychological impact. Many of the participants had had their VV for very long periods of time, often just "putting up with it'' for numerous years before seeking help. VLU 10 The impact of VLU on QoL differed within the group. For some there were no major issues and having a VLU was accepted as part of their current life, with the hope that it would heal eventually. For others, there was a far more significant effect. Pain was quite severe for some participants leading to a significantly reduced QoL. VLU appeared to have a significant psychological impact causing a high degree of distress for some. Abbreviations: Abdominal aortic aneurysm (AAA), Carotid artery disease (CAD), Peripheral arterial disease (PAD), Venous leg ulcer (VLU), Varicose veins (VV).  A total of 31 studies were included across the five reviews of existing qualitative research (6, 10-13). A short summary of the main themes to emerge for each condition is shown in Table 4.

Condition
Numbers of Citations  The themes from the first round of the clinicians' consensus study, as well as scales of identified PROM, were used to inform the framework analysis of the qualitative data.
Items from existing PROM were then mapped against emerging themes from the qualitative study, and the qualitative review synthesis for each condition, to explore which PROM items or scales captured themes deemed to be the most pertinent to patients. A triangulation approach was followed, whereby researchers evaluated whether the concepts were the same (agreement), offered similar concepts (partial agreement), were in contradiction (dissonance) or were not present (silence). An example of this triangulation approach is provided in Appendix 1.
The ePAQ-VAS development team used the findings from the triangulation for AAA, PAD, VV, VLU and CAD to develop themes for distinct sections relevant for each of F o r p e e r r e v i e w o n l y these vascular conditions. The primary qualitative data were used to create each item.
Items were then grouped into sections and within each section there were scales consisting of items that measured the same latent variable such as anxiety related to the diagnosis of AAA. The results of the clinicians' consensus study were considered to add further items to the relevant sections ( Table 5).
The items of ePAQ-VAS were arranged into four sections; generic, AAA, CAD and lower limb (LL) vascular conditions. A single LL section was developed as common themes were identified for conditions affecting the legs, regardless of whether the underlying pathology was venous or arterial. An inclusive approach to development was used and a comprehensive questionnaire was produced with 168 questions.

Most valid
There was little consistency in items that participants found difficult and no individual item was identified with which most participants had difficulty.
Discussion included the use of abbreviations, font size and contrast between text and background, response options and scales, electronic format versus paper format, relevance to patients and clinicians, the use of free text boxes and the language and wording used, when and how to use the skip button, repetition of items and subject matter and the possibility of emotional distress associated with questions about the possibility of deterioration or death.   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60   F  o  r  p  e  e  r  r  e  v  i  e  w  o  n  l  y   Insert figure 3 here   Based on the findings from the face validity exercise, and input from the Vascular PROM group, further revisions were made in an iterative process; culminating in the development of ePAQ-VAS. The structure of the questionnaire is illustrated in Figure   3. Fifty-nine items were eliminated for overlap; these include questions asking about common symptoms experienced across most vascular conditions. Five items were added based on suggestion from clinicians. Generic items for all respondents were presented in the first section and include questions about pain, altered sensation, The evidence used to develop each item in ePAQ-VAS is made explicit in Table 5 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59

Data sharing statement
No data are available.

Mobilitywalking speed
Positive

Social activities
Positive -

Independen ce/ burden
Positive

Patient and Public involvement:
The research question and output was developed in consultation with patients and The results will also be disseminated in relevant meetings and among patient groups.

Keywords:
Vascular disease, patient-reported outcome measures, quality of life, content validity, qualitative research, electronic questionnaire. The aim of this paper is to describe the stages undertaken to generate the items and conceptual framework of a new electronic personal assessment questionnaire for vascular conditions.

Design:
A mixed methods study: a clinician survey followed by quantitative systematic reviews to identify validated patient reported outcome measures (PROMs). A qualitative study of patients and systematic reviews of the qualitative evidence were used to develop this new PROMs. This was followed by a quantitative clinicians' consensus study and finally, a qualitative face validity study with patients.

Participants:
Vascular patients participated in the primary qualitative study and the face validity study. In the qualitative study fifty-five patients were interviewed and for the face validity nineteen patients gave feedback. Twelve clinicians completed the survey and thirteen completed two cycles of the clinicians' consensus study.

Results:
The items and scales in the electronic personal assessment questionnaire for vascular conditions (ePAQ-VAS) were generated based on the results of five systematic reviews evaluating existing PROMs for possible inclusion in ePAQ-VAS, five systematic reviews of qualitative evidence, a primary qualitative study involving 55 patients and clinicians' input. One hundred and sixty-eight items were initially generated; of which 59 were eliminated by the expert panel due to repetition. The instrument was divided into one generic, and three disease-specific sections (abdominal aortic aneurysm (AAA), carotid artery disease (CAD), and lower limb vascular conditions). In each section, items were grouped together into putative scales. Fifty-five items were grouped across eight scales; the remaining items were kept as individual items, because of relevance to service users.

Conclusions:
This multi-dimensional electronic questionnaire covers the most common vascular conditions. This is particularly important for patients presenting with mixed symptoms or multiple conditions. This tool captures symptomatology, HRQoL and other clinically relevant data, such as experience with services and co-morbidities.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  The burden of questionnaire is its main limitation; however, by providing strict skipping rules only relevant questions are presented to patients. The face validity study examined the clarity and relevance of the items, however, the comprehensiveness this PROMs was not assessed.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  Many generic and condition specific PROMs have been adopted to examine impact of vascular conditions on patients and measure outcomes. This is despite a lack of evidence that they have been developed and evaluated in-line with accepted guidelines; in addition, these instruments are rarely used or formally evaluated in routine patient assessment in day to day clinical practice. We conducted scoping searches and informal discussions with vascular clinicians to identify any existing PROMs, however, these preliminary stages in the research process suggested an absence of valid and reliable PROMs for use in vascular populations.
In this paper, we report the stages in developing an electronic personal assessment questionnaire for vascular patients (ePAQ-VAS). This includes: (iii) Developing the items within each section of ePAQ-VAS based on qualitative systematic reviews (9-12), and a primary qualitative study (13). (v) A face validity study with vascular patients to examine the clarity and relevance of the items within ePAQ-VAS.
The aim of these steps was to develop a single electronic instrument covering most vascular conditions in line with international guidance (3). The conceptual framework and items were developed in a way to ensure this assessment tool can be used in patients with mixed symptoms and multiple vascular conditions.

Methods
Clinicians involved in the care of vascular patients were invited to identify the common vascular conditions treated by vascular surgeons and vascular specialists.
They were asked to list the key issues, symptoms and the impact of these conditions on patients suffering with these diseases. Data from this round were used to inform qualitative evidence synthesis.

Analysis of the qualitative evidence
Qualitative data from the primary study and each of the systematic qualitative reviews were analysed separately. Framework analysis was used to analyse the interviews (15). This analysis includes five stages, the first involved familiarization by reading of the transcripts and reading the primary data.
-The second stage involved identification of a thematic framework; the thematic framework was based either on clinical opinion for areas with no valid PROMs, such as AAA and CAD, or a combination of clinical opinion and, when available, the scales of PROMs with good content validity. -In the third stage the data was coded and indexed by applying the thematic framework to the whole data set until saturation was achieved. Anindependent researcher checked all the themes that were identified, and differences in were discussed and adjusted involving a third senior author (GJ) -At the fourth stage, a framework matrix was created by arranging the data per the thematic references.
finally mapping and interpretation, including examining patterns within the data and associations with it.

Clinicians input and consensus exercise
Twenty-three clinicians involved in the care and management of patients with vascular conditions were invited to a survey to list the most common vascular conditions managed by them and to list the key issues, symptoms and the impact of these diseases on patients. Data from this round were used to inform qualitative evidence synthesis.
Different group of clinicians involved in the care of vascular patients were invited to a consensus study to score the relevance of items (questions) in the provisional version of ePAQ-VAS. In total, thirty clinicians including vascular surgeons, interventional radiologists, vascular nurses, physiotherapists and occupational therapists were invited. Participants were asked to rate the appropriateness of each question on a fivepoint Likert scale of "Strongly disagree" (=0) to 'Strongly agree' (=4). This process was repeated, and members of the clinicians' panel were presented with the aggregate findings of the previous round, and again asked to score each question. This process aimed to examine the relevance of each item from the clinicians' perspective and to identify any new items suggested by the clinicians (16).

Developing scales and items
The ePAQ development team (AA, EL, PP, GJ, SR) employed an iterative process, incorporating evidence from the systematic reviews, qualitative study, and the clinicians' consensus study. In line with the FDA guidance (3), items (questions) were  Interviews were audio taped, transcribed and analysed. A pragmatic approach was used for the analysis, with comments collated and presented back to the working group who made consensus decisions on revisions to ePAQ-VAS. Written consent was obtained from the participants.

Results
In total twelve clinicians completed the first survey and identified PAD, AAA, VLU, VV and CAD as the most common vascular conditions treated by them. They listed conditionspecific The most valid and reliable condition specific PROMs was VLU-QOL.
Abbreviations: Abdominal aortic aneurysm (AAA), Carotid artery disease (CAD), Peripheral arterial disease (PAD), Venous leg ulcer (VLU), Varicose veins (VV).      A total of 31 studies were included across the five reviews of existing qualitative research (6, 10-13). A short summary of the main themes to emerge for each condition is shown in Table 5.

Condition
Numbers of Citations   The themes from the first round of the clinicians' consensus study, as well as scales of identified PROMs, were used to inform the framework analysis of the qualitative data. Items from existing PROMs were then mapped against emerging themes from the qualitative study, and the qualitative review synthesis for each condition, to explore which PROMs items or scales captured themes deemed to be the most pertinent to patients. A triangulation approach was followed, whereby researchers evaluated whether the concepts were the same (agreement), offered similar concepts (partial agreement), were in contradiction (dissonance) or were not present (silence).
An example of this triangulation approach is provided in the supplementary material. The results of the triangulation study were only used to group symptoms together and avoid repetition. No items were deleted based on the triangulation. The ePAQ-VAS development team used the findings from the triangulation for AAA, PAD, VV, VLU and CAD to develop themes for distinct sections relevant for each of these vascular conditions. The primary qualitative data were used to create each item.
Items were then grouped into sections and within each section there were scales consisting of items that measured the same latent variable such as anxiety related to the diagnosis of AAA. The results of the clinicians' consensus study were considered to add further items to the relevant sections ( Table 6).
The items of ePAQ-VAS were arranged into four sections; generic, AAA, CAD and lower limb (LL) vascular conditions. A single LL section was developed as common themes were identified for conditions affecting the legs, regardless of whether the underlying pathology was venous or arterial. An inclusive approach to development was used and a comprehensive questionnaire was produced with 168 questions.
(See Figure 2 for an overview of the process to develop ePAQ-VAS).

Most valid Condition Specific PROMs
Qualitative study Qualitative Reviews

Scale
Clin icia ns' cons ens us Stu dy

Question Text
Gen eric Pai n 4 X X X X X X X X X X X X X Do you suffer with any pain?

Gen eric
Pai n 4 X --X X X X X X X X X X Use the below Image and click on body parts where you experience pain or discomfort. Gen eric Pai n -X --X X X X X X X X X X Please use your own words to describe this problem.

Gen eric
Pai n 4 X X X X X X X X X X X X X How often do you experience a significant amount of pain? Gen eric Pai n 4 ---X X X X X X X X X X How much do problems caused by pain affect your overall enjoyment of life?     The evidence used to develop each item in ePAQ-VAS is made explicit in Table 5, this table show whether the source for the item is the qualitative study, reviews or consensus study.

This instrument has been developed in line with FDA guidelines for developing
PROMs (   also need to include condition-specific items to fully capture the impact and clinically relevant information for each disease or condition. A further limitation is that the face validity study was not able to examine the comprehensiveness of ePAQ-VAS since it covered multiple conditions and it was difficult to expect from any of the patient groups interviewed to comment on diseases they have not experienced. Therefore, they only commented on the generic questions and the disease specific items relating to their condition.
In conclusion ePAQ-VAS is a multi-dimensional measure developed for use in a range of vascular conditions. It is a single electronic tool, covering most vascular conditions. This is important for those patients presenting with mixed symptoms or multiple conditions. The items in ePAQ-VAS can capture information about disease

Contributorship statement
JAM, GJ, AA, SCR designed the study. AA, PP, EL, SCR, GJ performed the analysis of the data and theme generation. ST and SN helped with consensus study and qualitative study recruitment. AA wrote the paper and all authors reviewed and edited the paper. All authors approved the manuscript.

Competing interests
Stephen

Mobilitywalking speed
Positive

Bending/ picking up things
Negative

Social activities
Positive -

Independen ce/ burden
Positive

Design:
A mixed methods study: a clinician survey followed by quantitative systematic reviews to identify validated patient reported outcome measures (PROMs). A qualitative study of patients and systematic reviews of the qualitative evidence were used to develop this new PROMs. This was followed by a quantitative clinicians' consensus study and finally, a qualitative face validity study with patients.

Participants:
Vascular patients participated in the primary qualitative study and the face validity study. In the qualitative study fifty-five patients were interviewed and for the face validity nineteen patients gave feedback. Twelve clinicians completed the survey and thirteen completed two cycles of the clinicians' consensus study.

Results:
The items and scales in the electronic personal assessment questionnaire for vascular conditions (ePAQ-VAS) were generated based on the results of five systematic reviews evaluating existing PROMs for possible inclusion in ePAQ-VAS, five systematic reviews of qualitative evidence, a primary qualitative study involving 55 patients and clinicians' input. One hundred and sixty-eight items were initially generated; of which 59 were eliminated by the expert panel due to repetition. The instrument was divided into one generic, and three disease-specific sections (abdominal aortic aneurysm (AAA), carotid artery disease (CAD), and lower limb vascular conditions). In each section, items were grouped together into putative scales. Fifty-five items were grouped across eight scales; the remaining items were kept as individual items, because of relevance to service users.

Conclusions:
This multi-dimensional electronic questionnaire covers the most common vascular conditions. This is particularly important for patients presenting with mixed symptoms or multiple conditions. This tool captures symptomatology, HRQoL and other clinically relevant data, such as experience with services and co-morbidities.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59 common (1, 2). It therefore makes sense to assess individuals with vascular disease holistically; investigating not only existing or potential manifestations of vascular disease, but also the impact of conditions on health-related quality of life. PROMs are questionnaires or instruments, designed to elicit information directly from the patient and can be used as part of such an assessment.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  Many generic and condition specific PROMs have been adopted to examine impact of vascular conditions on patients and measure outcomes. This is despite a lack of evidence that they have been developed and evaluated in-line with accepted guidelines; in addition, these instruments are rarely used or formally evaluated in routine patient assessment in day to day clinical practice. We conducted scoping searches and informal discussions with vascular clinicians to identify any existing PROMs, however, these preliminary stages in the research process suggested an absence of valid and reliable PROMs for use in vascular populations.
In this paper, we report the stages in developing an electronic personal assessment questionnaire for vascular patients (ePAQ-VAS). This includes: (iii) Developing the items within each section of ePAQ-VAS based on qualitative systematic reviews (9-12), and a primary qualitative study (13).
(iv) A consensus study with clinicians to rate the relevance of included items, and to add items to ePAQ-VAS based on the opinion of vascular surgeons, radiologists and nurses.
(v) A face validity study with vascular patients to examine the clarity and relevance of the items within ePAQ-VAS.
The aim of these steps was to develop a single electronic instrument covering most vascular conditions in line with international guidance (3). The conceptual  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60   F  o  r  p  e  e  r  r  e  v  i  e  w  o  n  l  y framework and items were developed in a way to ensure this assessment tool can be used in patients with mixed symptoms and multiple vascular conditions. Every patient to receive a unique voucher code along with their clinic letters. The code can be used to access and complete ePAQ-VAS at home or in the outpatient clinic using computers or other electronic devices.
The server of ePAQ is hosted and integrated with NHS N3-based informatics systems.
Other ePAQ questionnaires such as ePAQ-Pelvic floor and ePAQ-pre-assessment are in clinical use in different NHS hospitals. ePAQ Ltd. is an NHS spin-out technology company and the patient data collected by the company can be linked to the unique NHS number of each patient; and although there is a lack of integrated digital infrastructure in the NHS; the technology is available for future use to link records collected by different NHS providers

Methods
Clinicians involved in the care of vascular patients were invited to identify the common vascular conditions treated by vascular surgeons and vascular specialists.

Analysis of the qualitative evidence
Qualitative data from the primary study and each of the systematic qualitative reviews were analysed separately. Framework analysis was used to analyse the interviews

(15). This analysis includes five stages,
the first involved familiarization by reading of the transcripts and reading the primary data.
-The second stage involved identification of a thematic framework; the thematic framework was based either on clinical opinion for areas with no valid PROMs, such as AAA and CAD, or a combination of clinical opinion and, when available, the scales of PROMs with good content validity.
-In the third stage the data was coded and indexed by applying the thematic framework to the whole data set until saturation was achieved. Anindependent researcher checked all the themes that were identified, and differences in were discussed and adjusted involving a third senior author (GJ) -At the fourth stage, a framework matrix was created by arranging the data per the thematic references.
finally mapping and interpretation, including examining patterns within the data and associations with it.

Clinicians input and consensus exercise
Twenty

Developing scales and items
The ePAQ development team (AA, EL, PP, GJ, SR) employed an iterative process, incorporating evidence from the systematic reviews, qualitative study, and the clinicians' consensus study. In line with the FDA guidance (3), items (questions) were developed from the qualitative data using the following three steps: interpretation, translation, and triangulation of themes.  Interviews were audio taped, transcribed and analysed. A pragmatic approach was used for the analysis, with comments collated and presented back to the working group who made consensus decisions on revisions to ePAQ-VAS. Written consent was obtained from the participants.

Patient and Public involvement:
The research question and output was developed in consultation with patients and The results will also be disseminated in relevant meetings and among patient groups.

Results
In total twelve clinicians completed the first survey and identified PAD, AAA, VLU, VV and CAD as the most common vascular conditions treated by them.    standards (3,14). For instance, the review investigating VV PROMs (4) found some evidence for, and discussion of, content validity in relation to the Aberdeen varicose vein questionnaire (AVVQ) and suggest that it is the most appropriate existing condition specific measure for use in a VV population. However, item generation for this PROMs involved a literature review and assessment by clinicians of relevance of included items with no direct involvement of patients, therefore suggesting a deficiency in terms of content validity (17). The scales from these reviews were used to provide a framework for the systematic qualitative reviews and the primary qualitative study (5, 9-13).      A total of 31 studies were included across the five reviews of existing qualitative research (6, 10-13). A short summary of the main themes to emerge for each condition is shown in Table 5.
An example of this triangulation approach is provided in the supplementary material. The results of the triangulation study were only used to group symptoms together and avoid repetition. No items were deleted based on the triangulation.
The ePAQ-VAS development team used the findings from the triangulation for AAA, PAD, VV, VLU and CAD to develop themes for distinct sections relevant for each of these vascular conditions. The primary qualitative data were used to create each item.
Items were then grouped into sections and within each section there were scales consisting of items that measured the same latent variable such as anxiety related to the diagnosis of AAA. The results of the clinicians' consensus study were considered to add further items to the relevant sections ( Table 6).
The items of ePAQ-VAS were arranged into four sections; generic, AAA, CAD and lower limb (LL) vascular conditions. A single LL section was developed as common themes were identified for conditions affecting the legs, regardless of whether the underlying pathology was venous or arterial. An inclusive approach to development was used and a comprehensive questionnaire was produced with 168 questions.

Mobilitywalking speed
Positive