Article Text
PDF
Abstract
Objective To describe the development and clinimetric validation of the Advanced Practice Nursing Competency Assessment Instrument (APNCAI) through several evidence sources about reliability and validity in the Spanish context.
Design and setting APNCAI development was based on a multisequential and systematic process: literature review, instrument content consensus through qualitative Delphi method approach (a panel of 51 Advanced Practice in Nursing –APN– experts was selected) and the clinimetric validation process based on a sample of 600 nurses from the Balearic Islands public healthcare setting.
Methods An initial step for tool's content development process based on Delphi method approach of expert consensus was implemented. A subsequent phase of tool validation started from the analysis of APN core competencies latent measurement model, including exploratory and confirmatory techniques. Reliability evidence for each latent factor was also obtained. Items' scores were submitted to descriptive analysis, plus univariate and multivariate normality tests.
Results An eight-factor competency assessment latent model obtained adequate fit, and it was composed by ‘Research and Evidence-Based Practice’, ‘Clinical and Professional Leadership’, ‘Interprofessional Relationship and Mentoring’, ‘Professional Autonomy’, ‘Quality Management’, ‘Care Management’, ‘Professional Teaching and Education’ and ‘Health Promotion’.
Conclusions Adequate empirical evidence of reliability and validity for APNCAI makes it useful for application in healthcare policy programmes for APN competency assessment in Spain.
- Advanced Practice Nursing
- Clinical Competence
- Professional Competence
- Competency Assessment
- Psychometrics
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- Advanced Practice Nursing
- Clinical Competence
- Professional Competence
- Competency Assessment
- Psychometrics
Strengths and limitations of this study
The Advanced Practice in Nursing (APN) competency assessment instrument resulting of this study relies heavily in well-established APN competency maps from international institutions and it could contribute to support institutional initiatives of APN role delineation and implementation.
In many settings, no formal APN recognition exist but informal APN roles do, a validated tool for APN-specific competencies assessment capable of identification of those underlying roles could help to further formal APN roles implementation.
A coherent proposal based on professional APN competencies as shown in this paper could serve for the purpose of creating or revising educational programmes for formal APN role education.
Limitations of the study are related to the potential self-selection of respondents and lack of evaluation of criteria validity.
Introduction
The progressive spread of Advanced Practice Nursing (APN) worldwide1 has generated a whole range of approaches in the creation and development of this nurse role, not free of controversy and conceptual ambiguity. Development and establishment of new APN roles is a complex process2 that has resulted from a wide variety of reasons,3 although a key component is role definition, competency map delineation and role evaluation.4
It is noticeable both the heterogeneity of definitions, methodological difficulties through the different attempts at designing competency assessment methods in nursing,5 ,6 and the conceptual fragility of many instruments, assessment methods7 ,8 plus the difficulties in defining competence attributes.9 Reviews regarding the development of competencies in nursing do exist but none of them is specifically devoted to advanced practice.10–12
The progressive worldwide implementation of these advanced roles has led to a growing development of standards of practice and competency maps,13 but with many different conceptual approaches.14 Some recent efforts have been proposed to establish competency frameworks as a background to develop new APN roles15 ,16 or reviewing existing ones,17 essentially based on competency mapping and role delineation.18 Consequently, this new scenario needs to validate specific tools for APN competency assessment. Nevertheless, in certain cases, existing instruments are restricted to a particular context like the Scandinavian19 or the Japanese.20 Other instruments address very specific roles,21 ,22 or in some cases, these tools offer a more general proposal aimed at inferring competency domains that may be applied to the nursing profession as a whole.23 ,24 In the tools analysed, inconsistency in the validation process was detected in order to justify a new instrument development approach.
The aim of this study was to obtain adequate psychometric properties of the scores of an APN-specific competency assessment tool, based on international and multicontextual APN roles and standards of practice in a Spanish nurses' sample: the Advanced Practice Nursing Competency Assessment Instrument (APNCAI). This tool is aimed to operate independently of national/local legislative development, professional practice setting and/or regulatory context.
Methods
The development of the APNCAI was based on a multisequential and systematic process initiated with the literature review and subsequential expert consensus approach as pointed before. In order to develop a specific tool for assessing advanced practice competency, the research team started from an exhaustive worldwide literature review that focused on grey literature APN mapping documents, and classical health literature databases (details of this review have been reported elsewhere25). A final selection of 97 documents from different international settings was made by means of a pure content analysis, assisted by software MAXQDA. Results led to identify 17 core domains (clusters of related competencies) common to APN development at international level. Those initial domains included: Research, Clinical and Professional Leadership, Mentoring and Coaching, Collaboration and Interprofessional Relationships, Expert Clinical Judgement, Ethical and Legal Practice, Education and Teaching, Quality Management and Safety, Consulting, Care Management, Evidence-Based Practice, Professional Autonomy, Health Promotion, Communication, Cultural Competencies, Advocacy and Change Management. This original proposal contained a total of 212 items along the 17 domains. Following, a content validation process based on expert consensus was used in content process. Successive Delphi technique rounds were carried out, and as a result of this phase, 12 competency domains comprising a total of 54 individual competencies emerged, after two rounds of expert consensus plus a post-Delphi depuration approach. The details of this phase have been reported before.26
After the prior content validation, a pilot test phase with the initial instrument was conducted over a randomly extracted subsample of 50 nurses from a population of 3400 nurses belonging to the National Health System in Balearic Islands. These nurses belonged to acute care units in different hospitals and primary healthcare centres. All of them were selected if they had more than 15 years of clinical experience. This pilot phase was intended to achieve face validity, understanding of items and usability by cognitive interviews among this random sample of end users. Following this phase, some refinements in the length and clarity of some items were carried out.
Clinimetric tool validation process
Design
A proper reliability and validity content through clinimetric measurement approach27 was undertaken using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) for model generation28 to determine the underlying dimensional structure of APN core competencies.
Sample
Based on the 54 items in the preliminary version of the instrument, a minimum of 600 subjects were required, considering a statistical power of 0.80 and a confidence level of 95% to identify a difference of 0.03 units in Cronbach's alpha coefficient (hereafter,α).
Participants were included if they practised either in acute care units in hospital care, long-term care or in primary healthcare centres. No limits in years of experience were set as inclusion criteria to obtain a wide range of professional development situations to be evaluated by the instrument. A final sample of 600 nurses accepted voluntarily to participate from a population of 3400 (18%). Participants received an electronic password to access the online questionnaire, with guaranteed anonymity in the data processing. Informed consent was also provided.
Data collection
An online survey approach (Limesurvey) was used to collect data. A mail was sent to all potential participants to present the project, then asking them for voluntary participation and providing access to the final participants to the survey itself. The self-reported tool used a five-point Likert scale ranging from ‘never’ to ‘always’. Participants were asked to answer how often they performed every single competency statement in their current job.
Data analysis
The items in the preliminary version of the tool were submitted to descriptive analysis plus univariate and multivariate tests of normality. Internal consistency was analysed using Cronbach's α and interitem and item-total correlations. Data analysis consisted of a three-step sequence in a combination of EFA and CFA approaches. In the first round, principal component analysis (PCA) and a varimax rotation were carried out. Preliminary tests for justifying the analysis using the correlation matrix determinant, the Bartlett's test and the Kaiser-Meyer-Olkin (KMO) index were implemented. The Kaiser criterion was applied for extracting factors. Finally, a CFA was applied taking into account well-founded theoretical and empirical reasons, discarding possible capitalisation of random effects from EFA. The overall fit of the model was tested using penalised likelihood ratio χ2 (χ2/df), which is indicative of good fit with values <3, the Root Mean Square Error of Approximation (RMSEA) index and its 90% CIs, taking the value 0.06 as the cut-off for a good fit; the CFI (Comparative Fit Index) with a 0–1 range and the minimum value of a good fit set at 0.95 and, finally, the Standardised Root Mean Square Residual (SRMR), indicating good fit with values <0.08. A simple mean imputation method was used to replace missing values (<5% of total data). Cronbach's α values for all subscales were also computed. All analyses were performed with the statistical packages SPSS v.22.0 (SPSS., Chicago, Illinois, USA) and LISREL v.9.1 (SSI., Lincolnwood, Illinois, USA).
Results
The empirical sample was built up of 600 subjects, with a high predominance of women (79%), 94% had a university degree in nursing, while 3.8% had additional degrees in other disciplines. The percentage of nurses with official postgraduate education at master level was 13.3%, and with a doctorate degree 0.8%. For practice environment, hospital-related setting was 69.8%, primary healthcare 16.2% and other settings were 11.7%. Years of professional experience ranged from <1 to 43, with a mean of 15.4 (SD 10.9), and was normally distributed, as the Kolmogorov-Smirnov test was statistically non-significant (p=0.10). The heterogeneity of the sample of nurses provided a good basis that allowed to minimise the role-centred bias that an APNs-only sample could provide in self-reported scales.
The use of dimensionality reduction techniques requires the existence of an identifiable latent structure, tested by means of the determinant correlation matrix, that was very near to zero (1.5×10−17), the KMO that obtained an excellent value of 0.96 (values below 0.50 indicate that there is not adequate to reduce dimensions) and the Bartlett's test that was statistically significant (p<0.0001), indicating that the null hypothesis about the equality between the item correlation matrix and the identity matrix is rejected. The PCA extracted nine significant principal components that explained up to 65.79% of the variance. The initial factorial structure appeared to be closely related to what was expected from early process stages and in concordance with the key literature, resulting in a nine-factor model with a total of 54 items. In this phase, four items that showed poor performance, below 0.50 loading score, were deleted.
In order to avoid a possible capitalisation of chance of the exploratory factor structure model estimation, a CFA was implemented. The first model tested was exactly the nine-factor model that emerged during exploratory analysis. This nine-factor model showed problems with identification mainly due to the structure of two factors with only three indicators, ‘Cultural Competency’ and ‘Advocacy’ as well as factors eight and nine, whose three-item structure generated doubts regarding their fit in relation to the whole scale performance. Error variance of these two factors remained unidentified and, consequently, parameter estimation and the calculation of goodness-of-fit statistics were aborted by the programme. By mixing empirical and theoretical reasoning, these two factors were deleted. To consolidate this approach, all items of the model were revised in order to identify transversal main conceptual elements of advocacy and cultural competence in them if possible. From this approach, the 8th item of Factor 1, 3rd item of Factor 3, 2nd item of Factor 4, 3rd item of Factor 6 and the 4th item of Factor 8 were identified as containing recognisable elements from both eliminated factors. Also the new model showed that the first factor from the EFA now appeared as two distinct, recognisable independent Factors 1 and 2: ‘Research and Evidence-Based Practice’ and ‘Professional and Clinical Leadership’. Thereby, a new eight-factor latent structure with 44 items was estimated as shown in figure 1.
Factorial structure from confirmatory factorial analysis with correlation scores.
Labels and items' highest loadings for each factor are given in table 1.
Items' highest loading for each factor
Univariate and multivariate normality tests for all items did not detect special problems of non-normality, and the relative multivariate kurtosis value obtained an adequate value of 1.216. Results showed a good overall fit for the eight-factor model despite a statistically significant χ2 test (χ2=2275.97, df=839, p<0.0001), but with a penalised likelihood ratio below 3 (2.71), 0.05 for the RMSEA, 90% CI values of 0.05 to 0.06, CFI value very near to 1 (0.99) and finally 0.05 for SRMR, clearly below the cut-off criterion of 0.08. The factor loadings ranged from 0.50 to 0.87 for each dimension. The CFA loading matrix for the model estimated is shown in table 2.
Confirmatory factor analysis (CFA) loading matrix model of 8 factors
A full-scale reliability analysis was performed on the 600 subjects' response matrix database for the 44 items in the final version, which attained a reliability estimation of 0.96 using α coefficient. This value indicated a high degree of internal consistency of the full scale. Alpha coefficients for the final eight CFA dimensions were over 0.80. (Research and Evidence-Based Practice 0.92; Clinical and Professional Leadership 0.91; Professional Autonomy 0.90; Interprofessional Relations and Mentoring 0.84; Quality Management 0.84; Care Management 0.88; Professional Teaching and Education 0.81 and Health Promotion 0.85) Taking into account the corrected item-total correlation of each of the 44 items with the corresponding factors, all their correlation values performed over 0.40 (range 0.45 to 0.70).
Finally, the correlation matrix between latent factors was estimated as shown in table 3. The main findings at this stage are the strongest relationship between Factors 1 and 2. Apart from this, other strong interfactorial relationships appear, as seen below. The second and third strongest relationships appear in the concordance of Factor 5, with a core component of ‘Quality Management’, and Factors 1 (‘Research and EBP’) and 2 (‘Leadership’). The weakest correlation corresponds to Factors 8 and 2 (‘Leadership’ and ‘Health Promotion’, respectively).
Correlation matrix of latent factors
Discussion
APN core competencies have traditionally been an arguable question,29 ,30 with an ongoing discussion about defining the essential competencies for nursing practice and its difficulties for assessment methods and approaches for reflecting real nursing practice in clinical settings.
The aim of this study was to validate an instrument with a deep insight in the competency maps developed for APN roles by internationally competent and recognised institutions and/or organisations, to minimise this issue. Otherwise, recommended CFA was applied in order to evaluate interdependence of responses to the scale proposal.31
The main results of this study show how APN competencies cluster around eight competency domains, with a dominant factor based on ‘Research and Evidence-Based Practice’, followed by ‘Inter-Professional Relationship and Mentoring’ and ‘Care Management’, as the top loading factors, in terms of number of items. Likewise, a strong relationship between ‘Research and Evidence-Based Practice’ and ‘Leadership’ is shown, which is coherent with the previous literature, which points out a strong relationship between evidence-based practice and leadership in nursing,32 that acts as an essential element for evidence-based practice outcome indicators for the APN clinical environment,33 but it is essential for the existence of a dedicated source which acknowledge leadership roles, the provision of opportunities to act as a leader, key programmes for leadership development and the presence of specific personal attributes in APNs.34
This eight-factor latent structure comprised essential competency dimensions identified previously in literature. The model included elements of research capabilities and clinical leadership roles in conjunction with nursing practice supervision and assessment, as well as on implementation of patient care programmes, plus other factors dealing with treatments and intervention prescription, complex health problem identification and diagnosis, closely related to the professional autonomy field. Abilities for interprofessional relationship along with capabilities for performing as coach and mentor are other major concepts that fitted coherently in the latent analysis. The other factors fitted to the previous domain-based competency clusters identified in the previous process. Care management-related competencies fitted well in a single factor. Other consistent factors resolved around professional teaching and education.
There is also a consistent correlation between most factors, although all of them are distinct and capable of explaining large proportions of the variance in the data. A noticeable point is the weak correlation between Factors eight and two, with their main component being ‘Leadership’ and ‘Health Promotion’, respectively.
Further, there is a recognisable interpersonal relationship component in the competences included. Thus, collaborative practice between APNs and other healthcare providers (especially physicians) and away from previous supervisory relationships, has proved to potentially increase patient access to healthcare, optimise outcomes and improve communication, education and fiscal usage of healthcare facilities.35
Transitional care as a component of care management has also shown a key role in APN competencies. This area of professional development has been described in the literature as essential for increasing time to readmission, associated comorbidities or death, reducing readmissions and healthcare costs, while maintaining quality of life and functioning, by providing coordination and continuity of healthcare as patients transfer between different locations.36
Moreover, the leadership and knowledge brokering component37 understood as APN capabilities for improving evidence-based practice accessibility and consumption in their clinical context of reference, also emerges as an important area of competence (“Research and Evidence-Based Practice” with “Clinical and Professional Leadership” and “Professional Autonomy”), as has been previously reported in different APN areas of practice.38
One arguable point in the final eight-factor model proposal is the lack of a domain for ‘advocacy’, consistently identified as essential to APNs.39 The fact is that advocacy is underlying present in many of the other factors obtained and several competencies intrinsically encompass advocacy interventions.
Finally, the latent structure obtained is consistent with the main APN competency maps retrieved from international organisations, and those countries where APN has a consolidated development.40–43 Those competency mapping developments for APN closely related to Anglo-Saxon countries show that despite differences in APN role development approaches due to different healthcare system needs response, they share commons traits when working in identifying core APN competencies no matter the specific role described/implemented/assessed. The work presented in this paper also reflects those similarities even from a wider international perspective that included other countries’ APN role delineation approach.
Limitations
One main limitation of the study derives from the heterogeneity of the nurses’ sample that includes nurses with <5 years of experience, and others with no formal postgraduate degrees. A needed explanation of this limitation is based on conceptualising professional competencies, from the perspective that is understandable that no single competency belongs in exclusivity to the APN context. In fact, the competences evaluated may be present in generalist nurses, but, in different developmental gradients, aspect that can be detected by the scale by having several levels of evaluation of each competence.
In relation to professional competency conceptualisation, is arguably that all nurses, no matter they level of practice, could understand the essentials of every single competency with independence of the level of performance that a nurse could reach within a specific competency. From this perspective, a heterogenic sample of nurses provides a good basis that allows to minimise the role-centred bias that an APNs-only sample could provide in self-reported scales. The main argument here is that what the paper describes is essentially the validation of the latent model's structure, and further criterion and evidence validity is a needed step forward from the actual position.
Others limitations relate to the need to obtain criterion validity evidence and the building of normative scores to complete the psychometric validation process, that is not presented in this paper, and is a priority for future studies. Also, qualitative limitations, especially the potential bias derived from voluntary recruitment of the sample, of the nurse sample have to be taken into account for validation purposes in order to understand the external validity of the tool and its measurement capabilities.
Conclusion
This study provides a valid and reliable tool for the assessment of APN core competencies (APNCAI) that also serves to manage the selection and education of APNs, to develop the APN role in healthcare organisations or to formalise non-recognised advanced practice levels in some settings. Its multicontextual and global approach due to its main components being inferred through a process set in an international transversal cluster of competency related elements, provide the potential to be applied in different healthcare settings.
Moreover, our methodological study contributes to APN clarification, defining eight competency domains related to global APN delineated through a rigorous validation process. The resulting tool obtained could provide national organisations with a cornerstone framework for competency assessment with potential application in healthcare policies for APN role delineation and implementation; in direct clinical practice and management organisational levels through 360° competency assessment (that implies being assessed by superiors, peers and subordinates); in APN education programmes to assess learning outcomes for students' optimal performance and in APN research field to adequate essential competency areas in order to link them to best patient outcomes.
An international validation process, initially in the European context, is a necessary first step for further development of the tool. Assessment of the discriminant capabilities of the scale is also an interesting line of immediate development, exploring the sensitivity of the tool to identify potential APN roles under not formally structured APN services, or to compare level of competency between similar APN programmes in different healthcare settings. This initial effort has to be followed by an ulterior European first, and posterior international-level validation process, to overcome a full internationally applicable APN competency assessment tool.
References
Footnotes
Twitter Follow Miquel Bennasar-Veny @miquelbennasar
Contributors PS-F, JMM-A and JDP-G participated in the conception and design of the study and drafted the first version of it, as well as the manuscript. MB-V and JCF-D participated in the development of the study and also revised critically the draft of the manuscript. AS-A contributed decisively in the confirmatory factorial analysis part, all with a key intellectual contribution to the final version. All authors read and approved the final manuscript and approved its submission to the journal.
Funding The study was funded by Balearic Islands Nursing Council, Spain.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Ethics approval The study was approved by the ethical committee of the Balearic Islands University
Data sharing statement No additional data are available.