Objectives Studies have suggested that material deprivation is strongly associated with negative health outcomes, and lower usage of various levels of healthcare. We aim to analyse geographical access to emergency medical services (EMSs) and hospital emergency units by EMS in relation to deprivation in the Lisbon Metropolitan Area (LMA), Portugal.
Design This study estimates road network-based access times from the centroids of statistical sections (census block groups equivalent) to locations of EMS and hospital emergency services. Each statistical section has been linked to a Material Deprivation Index (MDI). A non-parametric analysis of variance (ANOVA) was undertaken to compare MDI-linked statistical sections in terms of access to emergency care. Geographical access analysis was conducted for 2018.
Primary outcome measure Road network-based access time (in minutes) for EMSs to statistical sections and then on to emergency units in hospitals.
Results Overall, 82.4% of the LMA population is located less than a 10 min drive from an EMS without transport, and 99.1% from an EMS with transport. Travel time from EMS with transport to hospital is potentially less than 20 min for 95.2% of the population. However, 63.1% of residents living beyond a 30 min threshold (total time from emergency call to hospital arrival) are in areas with very high MDI (18.8% in high MDI, 13.3% in medium MDI, 4.7% in low MDI, 0% in very low MDI). Kruskal-Wallis ANOVA confirms discrepancies in access times between better-off and poorer areas.
Conclusion Poorer areas experience worse geographical access to EMS and hospital emergency units. More research is needed to explore the quality of services and their outcomes, and to refine the analysis by focusing on specific vulnerable groups.
- geographical access
- medical emergency service
- emergency department
- spatial justice
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study addresses geographical access to both emergency medical services (EMSs) (emergency vehicles) and hospital emergencies simultaneously.
Potential travel time is estimated from the moment of the emergency call to arrival at the hospital, taking into account the location of EMS.
Deprivation Index and access times are measured at geographical level, not at the individual level.
Availability and levels of emergency medical structures were not taken into account.
The study did not consider time variability of traffic and service availability on a daily or weekly scale.
Spatial justice is concerned with equitable access to resources and opportunities. In many cases, improvements in living conditions and the distribution of services have been unequally shared across individuals, groups and geographical areas, with potential health consequences on vulnerable groups. There is indeed growing evidence of associations between residential location in socially deprived areas and a wide variety of health conditions,1–4 premature and preventable deaths5–7 and mental health8 and suicide.9
Socioeconomically disadvantaged areas are more likely to amplify conditions of material deprivation through a number of underlying interconnected mechanisms, leading to cumulative vulnerabilities4: (1) neighbourhoods that concentrate low-income groups with higher potential for poor health outcomes;10 (2) environments with high exposure to risk factors such as air pollution and road traffic11 and (3) environments that discourage healthy behaviours, for example, due to low access to green areas, shops and facilities.12
Since these disturbances are common, although not systematically associated to social deprivation,13 the geography of public healthcare facilities is normally expected to attenuate socio-spatial discrepancies by providing equal opportunities for at-risk population. This is not always the case, however, as the so-called ‘inverse care law’ prevails in many settings, meaning that deprived areas tend to have worse geographical access to healthcare.6 14 15 Contrasting with these conclusions, other studies have found that disadvantaged areas are actually better served.16
Low geographical accessibility to healthcare services has been found to be a strong factor of reduced utilisation. The well-known distance-decay effect has been identified in the usage of health services.17 This, again, tends to deepen the gap between better-off and deprived areas. While disparities in access to healthcare and their resulting health outcomes are increasingly well documented, most of the existing literature on travel times to hospital emergency services are based either on private car use or on travel time only between patients’ location and hospital.18 Access to emergency services not only depends on the distance to the nearest hospital, but also on how far the patients are from an emergency medical services (EMSs) that can take them to the final destination.19
In recent years, the Portuguese healthy system has been under pressure, as the 2008–2012 economic crisis led to a reduction in resources and facilities, with potential effects on health inequalities. Between 2010 and 2018, the Lisbon Metropolitan Area (LMA) has seen some changes in the location of emergency services (two closures, two relocations and one opening) (figure 1). Emergency services in Portugal are, since 1981, directed by the National Institute of Medical Emergency (INEM), which coordinates the Urgent Patients Orientation Centre that connects patients with EMS and hospital emergency units. EMS belonging to the fire stations and delegations of the Portuguese Red Cross (Cruz Vermelha Portuguesa—CVP) can be activated if the emergency call is located near them. EMS territorial bases therefore include INEM’s own bases, fire stations, CVP and emergency hospital units. More details are provided in online supplemental table 1.
The objective of this study is therefore to analyse geographical access to EMS and hospital emergencies across the LMA. This focus on EMS and hospital emergency services is important, first because of the relevance of time to survival, and second because successive political interventions have caused significant changes in hospital emergencies locations in the LMA, with unknown consequences to equality in access. Geographical access to EMS was analysed by the travel time of the nearest service in minutes, relative to the general population, and differentiated by levels of material deprivation. We hypothesise that potential travel times to EMS and hospital-based emergency services are higher from socially deprived areas than from other areas.
Data and methods
This cross-sectional, ecological study examines potential access to EMSs and emergency hospitals by EMS across deprivation levels, using network-based times estimates and non-parametric analysis of variance (ANOVA).
The LMA extends over 3015 km2 and accounts for 2.8 million inhabitants (27% of the national population). High dispersion and spatial fragmentation of the built environment is a leading cause of accessibility issues, which makes this case study all the more pertinent to other contexts. Analysing the region was preferred over the entire country due to the difficulty of using aggregate methods and indicators for both rural and urban regions, and because of LMA’s territorial specificities when compared with the rest of the country.
The shortest possible motorised travel time in minutes was estimated between EMSs, statistical sections, and hospitals using the Network Analyst tool of the ArcGIS V.10.2 program. Three types of routes were considered: (1) from the EMS to the initial point of care (EMS without transport); (2) the same routes, but using EMS with transport (differently located) and (3) the complete route between EMS with transport, the initial point of care (the statistical section) and the hospital using EMS with transport.
This study relied on four main data sources. First, the 2011 Census data—the most recent in Portugal—provided population data at the statistical section level. Statistical sections (or just sections—census block groups equivalent) are geographical areas of variable size depending on the built environment and the location (LMA average 66.4 ha, median 6.6 ha, n=4521). Each section has approximately 300 dwellings (table 1). In mainland Portugal, there are 17.337 sections (average area and population: 5.13 km2 and 579, respectively).
Second, the INEM provided the georeferenced location of all EMSs. Among the existing 10 types of EMS, only non-specialised and terrestrial EMS were included. Here, we distinguish between: (1) EMS without transport services (n=14), including VMERs (Resuscitation and Emergency Medical Vehicles, crewed by an emergency medical physician and a nurse, and advanced features of life support equipment) and MEMs (Medical Emergency Motorcycles, crewed by a single medical emergency technician and basic life support materials); (2) EMS with transport services (n=94), including AEMs (Basic Life Support Ambulances, crewed by two emergency medical technicians), ASs (Rescue Ambulance works in complementarity with other EMS and staffed by crew trained in emergency medical techniques) and SIVs (Immediate Life Support Ambulances, similar to an AEM, also performing resuscitation and including an emergency medical nurse in its crew). More details are provided in online supplemental table 1. Although travel times are estimated for EMS with transport and without transport separately, in practice the triggered EMS can be one of the two types, according to their availability at the moment of the emergency call.
Third, all addresses of hospitals with emergency services were collected and georeferenced for the year 2018. In this study, only intermediate (known as SUMC—Emergency Medical-Surgical Service, in Portugal) and more complex (SUP—Polyvalent Emergency Service) emergency hospitals were considered. The most basic emergency departments (known as Basic Emergency Service) operating in the primary health centre were not considered because of cause of their limited diagnostic and therapeutic capacity.
Fourth, the road network was based on TomTom (2012) GPS freeway and national roads networks, and on a previously modelled road network,20 which includes major urban roads and local roads network. The resulting network was modelled using Network Analyst in ArcGIS V.10.2. Travel times were estimated in minutes based on the legal top speeds of all infrastructures: freeways (120 km/hour), national roads (90 km/hour) and major roads (50 km/hour). Regarding local roads, speed was set at 30 km/hour based on comparison tests20 between travel time using GPS TomTom road network and the modelled network.
Patient and public involvement
No patients involved.
To assess area-level social vulnerability, a Material Deprivation Index (MDI) was created, drawing on three census data variables: population unable to read or write (per cent of individuals aged 10 and over), unemployment rate (per cent of unemployed among the active population) and substandard housing rate (per cent of houses without a toilet). This index was preferred to the EDI-PT (Portuguese version) —a more complex indicator adapted from the European Deprivation Index (EDI)21—because some of the variables used for building it are not free of charge at the statistical section level at the National Statistics Institute. The index used here has been associated to a variety of health outcomes and behaviours.3 8 9 22
All three indicators were standardised to the LMA means and then added together to form the Multiple Deprivation Index, in accordance to the z-score-based Carstairs and Morris’ method.23 Such composite indices have been widely used with some variations, and generally include variables related to education, housing condition, income, car ownership and/or employment.23 24 Here, we did not include car ownership, as its use in social deprivation indices is now increasingly debatable, first because the generalisation of cars has now made the indicator less powerful than it was for measuring poverty, and second because forced car ownership is frequently experienced as a financial burden that increases social vulnerability.
After calculating the MDI, statistical sections were divided into five MDI-based categories (or quintiles) (figure 2): Very Low (with MDI ranging from −1.43 to −0.56), Low (from −0.56 to −0.26), Medium (from −0.26 to 0.00), High (from 0.00 to 0.41) and Very High MDI (from 0.41 to 8.86). Very High MDI equates to the most deprived 20%; Very Low MDI equates to the least deprived.
Network-based access times between EMS, statistical sections and hospital emergency services were then estimated. The destination point corresponds to the nearest hospital unit with emergency services, with the journey measured in minutes. This is a common approach in studies of geographical accessibility to emergency services, primary care or ongoing treatment.17
Finally, MDI-based categories of sections were compared in relation to travel times. The analysis was performed using a Kruskal-Wallis test and a post hoc Dunn test, with statistical significance set at an alpha level of 0.05, to examine the differences between the various categories of MDI compared with access time, EMS/section access and EMS/section/hospital unit. This is a non-parametric version of ANOVA, designed to determine whether the medians of the categories are different. This test has the added advantage of being insensitive to both normality and variance heterogeneity assumptions; however, it is a less powerful test than the classic ANOVA or the Welch’s test.25 The Kruskal-Wallis test was chosen due to homogeneity of variance issues and normality issues when compared with first attempts based on a classical ANOVA and a Welch’s ANOVA approach, even after log-transformation of data. Homogeneity of variance was examined with Bartlett’s test (Bartlett’s K2=134.5, df=4, p value <0.001; K2=380.7, df=4, p value <0.001; K2=283.1, df=4, p value <0.001, respectively, for EMS without transport, EMS with transport from EMS to section and EMS with transport, from EMS to section and to hospital). A Shapiro-Wilk test was performed for normality: W=0.9, p value <0.001; W=0.8, p value <0.001; W=0.9, p value <0.001, respectively). This led us to choose the non-parametric test. In addition, a multiple comparison test was performed using Holm procedure26 in order to test the potential age structure effect as a covariate. Several variables have been tested as a covariate (old-age and young-age dependency ratios, per cent of residents under age 25, per cent of residents 65 or over). The reporting of this study follows guidelines from the STROBE statement (Strengthening the Reporting of Observational Studies in Epidemiology), except non-applicable items (6, 12-c, 13, 16-c).27
Global access times
Based on travel times, 82.4% of the population of the LMA has access to an EMS without transport within a 10-minute drive, 15.3% live between 10 and 20 min and 2.1% in the range between 20 and 30 min. The maximum access time is 34.9 min. Regarding EMS with transport, 99.1% is accessible within a 10-minute drive. Maximum access time is only 18.5 min. This is due to the greater number of EMS with transport than EMS without transport. Regarding full travel (from EMS with transport to section and from section to hospital), maximum access time is 44.7 min (more details are provided in online supplemental table 2). A total of 24 860 residents, corresponding to 0.9% of the total resident population of the LMA, live beyond the 30-minute threshold (full travel): 20.4% of them are aged 65 or over, and 15.8% are aged under 15. As expected, remote areas have longer access times (figure 3).
By MDI category
Based on deprivation levels, three findings can be highlighted. First, maximum access times are systematically higher in Very High MDI sections (most deprived areas) than in lower MDI sections. The biggest disadvantage of the most deprived areas is in accessing hospital emergencies via ambulance, rather than in accessing EMS. EMS without transport: MDI Very High 34.9 min and Very Low 23.3 min; EMS with transport: MDI Very High 18.5 min and Very Low 11.3 min. Regarding full travel (from EMS with transport to section and from section to hospital): MDI Very High 44.7 min and Very Low 27.3 min (figure 4). Maximum access times also show a gradient—although not always regular—from Very Low MDI sections to Very High MDI sections. The percentage of inhabitants living within 5 min of an EMS considered is higher in the most affluent areas than in the most deprived for the three routes (EMS without transport, EMS with transport and full itinerary): MDI Very Low 58.7%, 97.7% and 26.0%; and MDI Very High 39.6%, 85.6% and 15.0%, respectively.
Second, distribution of residents living beyond the 30 min threshold (full itinerary) shows that populations living in Very High MDI sections have worse access to emergency care than other deprivation levels: Very High MDI (63.1%/15 703 persons), High MDI (18.8%/4673), Medium MDI (13.3%/3308), MDI Low (4.7%/1176) and Very Low MDI (0%).
Third, the proportion of the population which experiences better access is always that with Very Low MDI, while the most deprived areas are almost always further from these services (figure 4). For example, 79.3% of people living in the least deprived areas (Very Low MDI) have a potential total transport time of less than 10 min (full itinerary), but only 63.8% of people living in the most deprived areas (Very High MDI) have an equivalent access time. Differences are apparent in almost every access time bracket. More details are provided in online supplemental table 3.
Analysis of variance
The Kruskal-Wallis test shows that there is a statistically significant difference between MDI-based categories, regardless of the type of itinerary: (1) EMS without transport (H(4)=99.866, p<0.01); (2) EMS with transport (EMS to section only) (H(4)=71.737, p<0.01) and (3) EMS with transport (full itinerary, from EMS to the nearest hospital) (H(4)=112.250, p<0.01).
Performing a post hoc Dunn test to the three itineraries allowed us to identify pairwise relationships between MDI-based categories and thus determine which of the medians are different from the others (online supplemental tables 4–9 provide a global view of descriptive statistics, including confidence intervals for means at a 95% level). The results do not fully support the hypothesis of unequal access to the EMS without transport, as there is no significant difference between, for instance, Low MDI sections (6.3±0.3 min) and Very High MDI sections (6.8±0.3 min). However, difference between Very Low MDI sections (5.1±0.2 min) and the remaining ones is highly significant, and so is the difference between Low MDI sections and Medium and High MDI sections (6.9±0.3 min).
The second set of relations (EMS with transport-based EMS sections itineraries) shows slightly shorter access times due to a stronger presence of available EMS with transport (n=94). Here, it becomes clear that there is a significant difference between Very Low MDI sections (2.2±0.1 min) and all other ones, while Low MDI sections (2.4±0.1 min) differ significantly from High (2.7±0.1 min) and Very High (3.1±0.2 min) sections. Differences are not all significant in the middle group composed of Low, Medium and High MDI sections; however, Very High MDI sections, which gather the poorest 20% of the sections, differ from almost every other.
Finally, based on the full itinerary with EMS with transport, three groups are clearly identifiable. First, Very Low MDI sections display the shortest times (7.5±0.3 min) and are significantly different from the others. Second, Low MDI sections are slightly less accessible (8.8±0.3 min) but still differ statistically from all other categories. Third, the Medium, High and Very High MDI sections (60% of the LMA population) statistically differ from the remaining ones, with longer access times to the nearest hospital: 9.5±0.4 min, 9.8±0.4 min and 10.0±0.4 min, respectively. The greater the MDI value, the longer the accumulated access time (EMS to section, and section to hospital). Again, more details are provided in online supplemental tables 4–9.
The findings are confirmed when adjusting for age structure using Holm procedure.26 Among age structure variables tested, the variable pc65 (per cent of residents aged 65 or over) provided the best adjustment. A 1% increase in the percentage of residents aged 65 or over results in a reduction of the total travel time, which varies according to the EMS considered. More details are provided in online supplemental tables 10–13.
This study contributes to the literature on socioeconomic inequalities and access to healthcare services. It shows that 82.4% of the population of the LMA is within a 10 min drive from an EMS without transport and 99.1% from an EMS with transport, while total travel time from EMS with transport to hospital is less than 20 min for 95.2% of the population. However, when deprivation levels are taken into account, the worst access times are associated with the highest deprivation levels as the proportion of people living beyond a 20 and 30 min threshold (full itinerary, EMS with transport) is greater in Very High MDI sections than in any other. The relationship between MDI and access times was confirmed through non-parametric ANOVA.
These findings extend current knowledge on the identification of small areas with lower access to healthcare and their association with deprivation levels, by relying not only on the location of people and hospitals, but by taking into account the location of EMSs and estimating the total travel time needed from the call to arrival at the hospital. This is important because the incorporation of the location of EMS in travel times can substantially reinforce or attenuate access discrepancies. EMS bases offer higher flexibility than hospitals, and their provision can more easily be adapted to the distribution of needs across space. The existence of discrepancies in total travel times may not be without consequences, first because in many emergency cases survival is time sensitive,28 and second because with increased distance the utilisation of emergency units diminishes, particularly when deprivation levels are accounted for.29
Our findings are in accordance with several studies showing associations between deprivation areas and low geographical access to different types of healthcare services. In the Portuguese context, a study that covered a wide range of health services, at the national level and the scale of the municipality, also found evidence of injustice in access to healthcare, which is more scarce in the most impoverished areas.6 Other studies found that the areas with the highest deprivation level were those with the worst geographical access to general practice and acute hospital,30 perinatal healthcare and hospitals.4 A few other studies—not focusing on emergency services—concluded differently, however, as healthcare provision appears to be more equally distributed in some contexts.16
Reduced access time to both the nearest EMS and the nearest hospital is important to ensure an efficient and inclusive healthcare service. In that sense, geographical access to EMS is as crucial for the survival rates of socioeconomically disadvantaged individuals as access to hospitals, either for urgent cases, chronic disease patients who need ongoing treatment or those who use hospital services regularly such as patients with diabetes.31 Maximising geographical access to EMS in highly deprived areas could be achieved by a shift towards the logic of the positive care law in the organisation of EMS.15 Any improvement should begin from enhancing geographical access to the first point of contact with EMS and hospital emergency services for groups at risk. This points to the need for a more comprehensive research on the location of EMS that takes into account socio-spatial inequities.
In Portugal, the benchmark geographical access time to any unit with emergency service is 60 min.32 However, this can be an overestimate as it does not include time spent on scene before transfer to hospital. In 2007, the Technical Commission for Support to the Emergency Requalification Process proposed 30 min as the benchmark for any unit within the emergency services and 45 min for an emergency SUMC or SUP. It was also argued that, in urban areas, 90% of prehospital service responses should ideally occur within 15 min.33 While our study shows that the LMA context fits these recommendations, we also argue that any recommendations should take into account the geographical distribution of social strata, instead of merely using general population levels.34
This study has several limitations, the first of which is related to locations of both demanders and providers. The use of population-weighted centroids of statistical sections raises the well-known aggregation error bias issue,35 although it is a widely accepted method in the absence of more accurate data.34 EMS locations are more problematic since they can be activated when they are off base. EMS are likely to be located fairly close to hospitals at any given time, so it is possible that access is underestimated in the case of poorer neighbourhoods. Moreover, this study assumes geographical proximity as the single factor determining which hospital is the destination. Emergency services utilisation can be modified by administrative boundaries. It might happen that in some cases hospital catchment areas do not coincide with what would be network-based areas of influence. A possible consequence is that travel times may be underestimated in our analysis. However, emergency services are significantly less constrained within such boundaries than primary care services.
The second limitation relates to the estimated travel times reflecting potential geographical access, first because the time spent travelling to the hospital by the EMS does not account for the time initially dedicated to on-site care by the EMS, and second because daily and weekly variability of traffic is not considered here. Finally, this study did not consider the actual availability of emergency medical hospitals to receive patients, as well as the availability of medical specialties in each structure. More research should consider overcoming these limitations.
Further research is needed to improve accessibility measurements and to acknowledge spatial injustice issues related to specific medical causes. While our study includes all potential causes resulting in travel to an EMS, it does not distinguish between them. Spatial injustice patterns may be different for different health conditions. Deepening the discussion of spatial justice in health would also include assessing the delivery of emergency services (outputs) and their results (outcomes). Future research should focus on some vulnerable groups such as older adults and migrant groups. In the context of both ageing populations and increasing migration (migrant populations are often associated with poorer health), the availability of pre-emergency care will be increasingly under pressure.
This study contributes to a better evaluation of inequalities in access to healthcare and complements previous studies conducted in Portugal.1 6 36–38 To our knowledge, no previous study specifically addresses travel times to EMS and to hospital emergency service by EMS in Portugal. The analysis presented here gives support to the notion that space can act to reinforce of health inequalities. It thus becomes increasingly necessary to rethink EMS distribution in a way that counteracts spatial injustice and improves geographical access to the first point of contact for emergency services in the most impoverished areas. This study draws attention to the importance of considering the socioeconomic characteristics of the population when planning for EMS location. This is particularly relevant for households without a private vehicle that depend exclusively on the availability of EMS to get to the nearest hospital.
The authors are grateful to the Portuguese National Institute of Medical Emergency for graciously providing the data. The authors are also grateful to Marcos Correia for providing the road network that was used in the study and Archie Davies for the linguistic reviewing of the manuscript. The authors also thank Luís Henrique Gama Dore de Araújo and Bruna Bronhara for the statistical review of the methods and findings. Finally, the authors are grateful to the reviewers for their helpful comments.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Contributors KSdNS conceptualised the study and undertook the literature review, collected the data, built the database and analysed the general results. MP designed the statistical analysis and analysed the results. At different stages KSdNS and MP were involved in the development of the methods, data analysis and interpretation. KSdNS led the drafting of the manuscript. MP reviewed the manuscript, and KSdNS revised it. All authors approved the final version.
Funding This work was supported by the Centre of Geographical Studies of the Institute of Geography and Spatial Planning at the University of Lisbon funded by the Fundação para a Ciência e aTecnologia (FCT, I.P.), under the grants «UIDB/GEO/00295/2020» and «UIDP/GEO/00295/2020»; and the Coordination for the Improvement of Higher Education Personnel, a Foundation within the Ministry of Education in Brazil with the research project Health and Spatial Justice: The Geography of the Emergency Services in the Lisbon Metropolitan Area approved under the agreement number 1685-13-9.
Map disclaimer The depiction of boundaries on the map(s) in this article does not imply the expression of any opinion whatsoever on the part of BMJ (or any member of its group) concerning the legal status of any country, territory, jurisdiction or area or of its authorities. The map(s) are provided without any warranty of any kind, either express or implied.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Access time estimates and Material Deprivation Index categories are available from the Figshare repository, DOI: 10.6084/m9.figshare.13153952. The sections IDs have been removed from the dataset; they can be provided upon request. Other data are available upon reasonable request. We are however unable to provide the EMS dataset, as this is a proprietary dataset (National Institute of Medical Emergency-INEM). We also are unable to provide the road network dataset, as it was graciously provided by our colleague Marcos Correia.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.