Article Text
Abstract
Objective To study the psychometric properties of the Georgian version of the Safety Attitudes Questionnaire short version.
Design Cross-sectional study.
Setting Three Georgian hospitals.
Participants Personnel of participating hospitals (n=305 responses, estimated response rate 30%).
Interventions None.
Primary and secondary outcome measures Psychometric properties (model fit, internal consistency, construct validity, convergent and discriminant validity) of the instrument, factor structure derived from the data.
Results The Georgian version of Safety Attitudes Questionnaire demonstrated acceptable construct validity and internal consistency (Cronbach’s alpha 0.61–0.91). Three factors, Teamwork Climate, Safety Climate and Working Conditions, had limited convergent and discriminant validity. Confirmatory factor analysis with the original six-factor model resulted in limited model fit (χ2/df=2.14, root mean square error of approximation (RMSEA)=0.06, goodness of fit index (GFI)=0.83, CFI=0.88, TLI=0.86). Exploratory factor analysis resulted in a modified four-factor model with satisfactory model fit (χ2/df=2.09, RMSEA=0.06, GFI=0.88, CFI=0.93, TLI=0.91).
Conclusions The Georgian version of the Safety Attitudes Questionnaire (short version) demonstrated acceptable psychometric properties, with acceptable to good internal consistency and construct validity. While the whole model had limited fit to the data, a modified factor model resulted in good model fit. Our findings suggest the dimension Working Conditions has questionable psychometric properties and should be interpreted with caution. Other two correlated dimensions Teamwork Climate and Safety Climate share considerable variance and may be merged. Overall, the instrument can provide valuable information relevant for advancement of patient safety culture in Georgian hospitals.
- health & safety
- quality in health care
- statistics & research methods
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
First study to validate a Georgian version of the Safety Attitudes Questionnaire (short version).
A comprehensive analysis of the survey instrument’s performance, including exploratory and confirmatory factor analysis.
Multiprofessional sample from multiple hospitals.
Study findings are limited by the study sample, which included three general hospitals.
Introduction
Over the past decades, overwhelming evidence has been accumulated suggesting that patient safety is an ongoing challenge for modern healthcare systems. Cultivating the culture of safety in healthcare organisations has been recommended to enable better communication and open exchange, to learn from errors, eventually leading to better patient outcomes.1 The recent report by the Organisation for Economic Cooperation and Development (OECD), the Economics of Patient Safety2 analysed the state of the research and its implications on a larger, national level. Among the key messages and conclusions, the report underlines the importance of placing patient safety among national priorities, and establishing positive patient safety culture at the organisation level. Ensuring high quality and safe healthcare services for all citizens, in line with these recommendations, is also among the current priorities of the Georgian government.2 3 The healthcare services in the country are largely provided by private organisations which are increasingly required by state regulatory organisations and funding bodies to establish processes and systems to ensure improvements in patient safety.4 Thus, in order to analyse and develop the culture of safety, Georgian healthcare organisations require valid instruments to measure safety culture in local environments.
The Safety Attitudes Questionnaire short version (SAQ-S),5 originally adapted from the aviation industry to be used in US hospitals, is among the most frequently used instruments for measuring safety culture internationally.6 It has been translated into several different languages and validated in many countries.6–14 Overall, validation studies using SAQ-S have reported acceptable psychometric properties, indicating that the instrument may be relevant for measuring and promoting patient safety culture in different healthcare settings.
In this study, we aimed to evaluate psychometric properties of a Georgian version of the Safety Attitudes Questionnaire short version (SAQ-S-GE), including internal consistency, convergent and discriminant validity, fit to original factor model and possible alternative factor structure.
Methods
Setting and data collection
The data for this cross-sectional study were collected as part of the study project Patient Safety Culture in Georgian Healthcare (PaSCu.Ge), which aimed to establish a baseline evaluation of patient safety culture in local hospitals using translated and adapted versions of internationally well-studied instruments, the Hospital Survey on Patient Safety Culture15 and Safety Attitudes Questionnaire.5 The psychometric properties of the former are presented elsewhere.16 Using the SAQ-S-GE, data were collected in three Georgian hospitals in two cities between June and August 2017. Hospital employees could complete the survey electronically or on paper. Hospital representatives were trained to act as local study coordinators and facilitate employee participation.
Patient and public involvement
Representatives of patient and public groups were not involved in the study design and implementation. Main findings of the study will be made publicly available online (in both Georgian and English).
Sample
Two of the three participating hospitals have 100–150 hospital beds and the third has <50 hospital beds. All three are for-profit multiprofile hospitals, with an estimated total of 1000 employees who met the inclusion criteria. All personnel of the three participating general hospitals, employed for more than 1 month, were invited to participate. Before completing the survey, all participants were informed that participation was voluntary and anonymous, and provided informed consent.
Measure
Safety Attitudes Questionnaire short version (SAQ-S)5 17 consists of 36 items, 31 of which are grouped into six dimensions. All 36 items of the instrument measure participants’ agreement (from 1=strongly disagree to 5=strongly agree) to various patient safety-related statements on a five-point Likert scale. In this study, we also included the outcome item Patient Safety Grade from another widely used instrument on patient safety culture, the Hospital Survey on Patient Safety Culture.15 The item asks for an employee evaluation of patient safety on a five-point quality scale (from 1=failing to 5=excellent). In addition, the questionnaire included the demographic information on study participants (ie, department, profession, gender, tenure).
The original version of the SAQ-S was translated from English to Georgian by a native speaker with experience of working in Georgian healthcare. The translated version was adapted to the Georgian healthcare context without changing the overall structure of the instrument. Next, the Georgian version was back-translated to English by a professional translator. The discrepancies with the original version were discussed by the research team (NG, AH and TM) and necessary revisions were made. We asked five Georgian professionals (healthcare researchers and managers, physicians and nurses) who were not otherwise associated with the study to do a cognitive pretest of the revised version and to provide feedback on the content and language. Based on the results of the pretest, we were able to establish the final Georgian version of the questionnaire used in this study (SAQ-S-GE). In order to support comparability of the results, we made sure to maintain the overall composition and item wording of the original SAQ-S. The SAQ-S-GE is available on request from the corresponding author.
Analysis
Data processing and preliminary analysis
Before the analysis, negatively coded items were reverse coded, so that higher scores correspond to more positive safety culture. Descriptive analyses, as well as analyses of acceptability, were conducted using the complete sample. It has been shown that by means of imputing the missing answers, a considerable part of the sample may be made available for the analysis sensitive to missing values.18 However, in order to maintain high data quality, we excluded cases with more than 10% missing answers before imputation. The remaining missing values were imputed using the expectation maximisation algorithm.
Descriptive statistics
Mean scores for SAQ-S-GE dimensions were calculated by averaging the corresponding items. We calculated means and SD and the percentage of positive responses (scores 4 and 5) for each item and dimension.5
Acceptability
To evaluate the acceptability of the questionnaire, we calculated the percentage of missing answers on individual items and complete dimensions. We considered floor and ceiling effects (ie, the percentage of lowest and highest available answers, respectively), as an indication of the instrument’s performance at the extremes of the measured construct. For dimensions, we considered a floor or ceiling effect of <15% acceptable.17
Internal consistency, construct validity, convergent and discriminant validity
As an indication of internal consistency of the instrument, we calculated Cronbach’s alpha for each dimension. Cronbach’s alpha ≥0.7 was considered good.19 20 We evaluated Spearman’s correlations between the SAQ-S-GE dimensions, as well as correlation with the additional single-item outcome variable Patient Safety Grade, as preliminary analysis of construct validity. Because all dimensions are considered to be measuring constructs related to patient safety, we expected to find low to moderate positive correlations. However, excessive correlation between dimensions (>0.85) could indicate possible collinearity.15 19
Additionally, we evaluated the convergent and discriminant validity of the SAQ-S-GE.21 As an indication of convergent validity of a dimension, we calculated the average variance extracted (AVE) and expected it to be >0.5. For divergent validity, we used the Fornell-Larcker criterion21 and expected a higher square root of AVE (√AVE), compared with the highest correlation with other factors.
Exploratory factor analysis
Before conducting the factor analysis, we evaluated if the data were suitable for the analysis. We used Kaiser-Meyer-Olkin (KMO) and measure of sampling adequacy (MSA) (>0.7 desired, >0.9 perfect) to evaluate if the sample of items and each individual item were adequate for factor analysis.19 A significant p value (<0.05) of Bartlett’s test of sampling adequacy would indicate that it is possible to extract more than one factor.19
To explore a possible alternative factor structure based on our data, we conducted exploratory factor analysis (EFA) using the 31 items of the original six dimensions of SAQ-S. We used maximum likelihood algorithm for factor extraction with Varimax orthogonal prerotation and Promax oblique rotation to aid with interpretation of the factor model.20 The number of factors extracted was guided by scree plot inspection and the Kaiser criterion (eigenvalues>1). We considered factor loadings ≥0.4 significant and cross-loading <0.4 acceptable.19 20 We evaluated the similarities and differences between the EFA-based modified factor structure and the original model.
Confirmatory factor analysis
Confirmatory factor analysis (CFA) was used to evaluate how well the data fit the original factor model. The hypothesised model of SAQ-S-GE is presented in online supplementary appendix 1. We used the following indices and benchmarks: normed χ² (χ²/df ≤2.5), root mean square error of approximation (RMSEA≤0.07), goodness of fit index (GFI>0.90), comparative fit index (CFI≥0.90) and Tucker-Lewis Index/non-normed fit index (TLI>0.90).5 19 20 We analysed the fit of complete model, as well as that of each of the six original dimensions. Finally, we evaluated the fit of the EFA-based modified model. All analyses were done using SAS V.9.4.
Supplemental material
Results
Study sample and descriptive statistics
A total of 305 questionnaires were collected from three participating hospitals, resulting in an estimated response rate of 30.5%. Twenty-one participants (6.9%), all from one hospital, used the online questionnaire. Most participants indicated having direct contact with patients (83.3%) and no managerial functions (61.0%). Descriptive characteristics of the sample are presented in table 1.
Percentage of missing answers per item was 6.8% on average, with a minimum of 2.3% (“Working here is like being part of a large family”) and a maximum of 13.1% (“I receive appropriate feedback about my performance”). The dimension Stress Recognition had a floor effect >15%, while all other dimensions demonstrated a ceiling effect of >15%. Mean values and percentages of positive responses, as well as corresponding standard errors for all SAQ-S-GE dimensions, each individual item and the single item Patient Safety Grade are presented in table 2. The table presents all original 36 items and the corresponding six factors according to the guidelines of the Centre for Healthcare Quality and Safety of the University of Texas (available at https://med.uth.edu/chqs/survey).
Internal consistency and construct validity, convergent and discriminant validity
After removing 42 cases with more than 10% missing answers on any of 31 SAQ-S-GE items and imputing the remaining missing values, 263 questionnaires were available for further analyses. The dimensions of the SAQ-S-GE demonstrated good internal consistency with Cronbach’s alpha ≥0.7, with the exception of the dimension Working Conditions (α=0.61). All interfactor correlations were statistically significant, except for the correlation between Stress Recognition and Working Conditions. Most dimensions correlated positively with each other, except for Stress Recognition, which had a negative correlation with all other dimensions. Convergent validity of the three out of six dimensions, Teamwork Climate, Safety Climate and Working Conditions, failed to reach the required benchmark of 0.5. Also, √AVE of these three dimensions was much lower than the highest correlation with other factors, demonstrating limited discriminant validity. Another three dimensions, Job Satisfaction, Stress Recognition and Perceptions of Hospital Management, showed good reliability with good internal consistency and convergent validity, and acceptable divergent validity, with Job Satisfaction having slightly less √AVE compared with the highest correlation. Table 3 presents the results of validity analyses of the original six dimensions.
Evaluating fit of the data to original model, and exploring an alternative model
KMO test returned 0.89, while the average MSA for the individual items was 0.86 and varied between 0.59 and 0.95. Bartlett’s test was highly significant (p<0.0001), further indicating that the sample was adequate for factor analysis.
EFA resulted in a modified four-factor model with 21 items. Ten items were removed from this model because of either low factor loadings (<0.4) or high cross-loadings (>0.4). Two dimensions, Teamwork Climate and Safety Climate, were merged to form a combined dimension of Teamwork and Safety Climate. Three original dimensions, Job Satisfaction, Stress Recognition and Perceptions of Hospital Management, remained in the model, retaining all or most of the original items. The items from the dimension Working Conditions were mostly removed from the model. The EFA-based four-factor model is presented in table 4.
In CFA, we evaluated the fit of our data to the original six-factor model.5 While the whole model hardly satisfied set criteria for a good fit, individual dimensions had better fit indices. Finally, we checked the fit of the EFA-based four-factor model to the data using CFA, which resulted in acceptable fit indices. Results of all CFA, together with considered thresholds for acceptable fit, are presented in table 5.
Discussion
In this study, we evaluated the psychometric properties of SAQ-S-GE by analysing its acceptability, internal consistency, convergent and discriminant validity, as well as the fit to the original model and an alternative factor structure. We were able to identify dimensions with satisfactory properties and dimensions that may need further improvements and/or study.
Overall, the SAQ-S-GE was well accepted, with acceptable percentages of missing answers. Some studies using SAQ reported much lower (<2%) missing rates,5 7 14 while our findings are more in line with European studies, reporting up to about 10% missing.9 10 22–24 Further, we observed a significant ceiling effect in most dimensions and items, indicating that the instrument may not be effectively distinguishing the measurements at the higher end. A recent study using Rasch analysis similarly found considerable ceiling effect in all dimensions of the SAQ-S.25 Moreover, in the recent study conducted in Georgian hospitals with another safety climate instrument, Georgian version of Hospital Survey on Patient Safety Culture, we have also found considerable ceiling effect in most dimensions.16 This effect may be explained at least partially by social desirability bias pushing responses towards the positive end. Interestingly, the only dimension in our study demonstrating a floor effect was Stress Recognition. This dimension stood out in further analyses as well.
Our analysis of internal consistency obtained acceptable to good results, with only the dimension Working Conditions showing a low Cronbach’s alpha. This is in line with other studies using the SAQ that reported lower internal consistency for this dimension.12 13 23 24 Most items from this dimension had relatively high rates of missing values and were eventually removed from the alternative factor model, further indicating possible problems with validity or stability of the dimension. At this stage, the scores from this dimension should be interpreted with caution.
All dimensions except Stress Recognition were positively associated with each other, and with the outcome item Patient Safety Grade, reinforcing the validity of the instrument to measure a common underlying construct—Patient Safety Culture. The dimension Stress Recognition, however, was negatively associated with most other dimensions, which is well in line with findings from other studies using the SAQ.5 9 10 23 24 26 Taylor and Pandian26 called this dimension ‘a dissonant scale’ and recommended separating it from the instrument. Indeed, as many authors have pointed out, this scale asks participants for a self-evaluation while all remaining items refer to behaviours of others (ie, team, management or organisation). In spite of the described unexpected performance, items included in this scale, representing recognition of the effect of stress on performance, are without a doubt important for establishing a better culture of safety. Thus, stress recognition should be further measured and developed in healthcare organisations.
Our analysis of convergent and discriminant validity revealed problems with three dimensions—Teamwork Culture, Safety Culture and Working Conditions. The former two seem to have limited validity because of high intercorrelation, which was reaffirmed in EFA by merging these two dimensions together. Similarly, in most studies using SAQ-S, the correlation between these two dimensions was moderate to high, including the original study by Sexton et al 5 reporting within-area correlation of 0.94. This could indicate overall association between these two dimensions, not specific to our study. The third dimension, Working Conditions, not only demonstrated poor validity but was also mostly removed from the alternative model. Similarly, EFA conducted by Smits et al, 11 using an ambulatory version of the instrument, resulted in a five-factor solution, with Working Conditions not being presented in the model.5 Overall, except for the items associated with the dimension Work Conditions, the SAQ-S-GE adequately measures the underlying constructs.
Limitations
Our sample was limited to three general hospitals from two cities in Georgia while there are many other similar hospitals in the country. Thus, the findings should not be directly generalised to other healthcare settings in Georgia. Another limitation of the study was the relatively small sample size, due to which we were not able to conduct a split-sample validation. Thus, our findings should be tested in future research using a larger independent sample. Further research could also establish the ability of the instrument to measure change over time, which was not allowed by the cross-sectional study design.
Conclusions
The SAQ-S-GE demonstrated adequate psychometric properties, with acceptable to good internal consistency and construct validity. While the original six-factor model had poor fit to the data, we demonstrated an alternative factor model with acceptable model fit, indicating one problematic dimension and most dimensions being relatively stable and thus suitable for further studies. Until these findings are cross-validated in future studies with larger sample size, we argue for using the instrument in its full form, but recommend caution while interpreting the data on Working Conditions. As in most other studies, the dimension Stress Recognition was dissociated from the remaining instrument, while simultaneously demonstrating good psychometric properties. This dimension may require further investigation. The SAQ-S-GE can provide valid and useful information to further patient safety culture in Georgian hospitals.
Acknowledgments
We would like to thank the hospital management for agreeing to participate, the local study coordinators for their tireless effort in data gathering and all hospital employees who managed to find time to fill in the surveys.
Footnotes
Contributors The study was designed by NG and TM, in cooperation with AH. The hospital representatives facilitated collection of the data under the supervision of NG. Data processing and analysis was carried out by NG. The manuscript was prepared by NG in cooperation with TM, AH and NE. The final version of the manuscript has been approved by all authors.
Funding This work was supported by the German Academic Exchange Service (Funding programme: 57214224, to NG) and the BONFOR programme of the University Hospital Bonn (grant number: 2017-5-01).
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The research ethics committees of the University Hospital Bonn (#117/17) and the University of Georgia (#05-13779) approved the study.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available. Data from the PaSCu.Ge study will not be made available in the public domain to preserve data security. Data will be stored in accordance with the national and regional data security standards.