You are here

  • Home
  • Archive
  • Volume 6, Issue 1
  • Markers of achievement for assessing and monitoring gender equity in translational research organisations: a rationale and study protocol

Article Text

Article menu

Download PDFPDF

XML
Health policy
Protocol
Markers of achievement for assessing and monitoring gender equity in translational research organisations: a rationale and study protocol
  1. Pavel V Ovseiko1,
  2. Laurel D Edmunds1,
  3. Linda H Pololi2,
  4. Trisha Greenhalgh3,
  5. Vasiliki Kiparoglou4,
  6. Lorna R Henderson4,
  7. Catherine Williamson5,6,7,
  8. Jonathan Grant8,
  9. Graham M Lord6,7,9,
  10. Keith M Channon1,4,10,
  11. Robert I Lechler11,
  12. Alastair M Buchan1,4,10
  1. 1Medical Sciences Division, University of Oxford, John Radcliffe Hospital, Oxford, UK
  2. 2National Initiative on Gender, Culture and Leadership in Medicine: C-Change, Brandeis University Women's Studies Research Center, Waltham, Massachusetts, USA
  3. 3Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  4. 4NIHR Oxford Biomedical Research Centre, Joint Research Office, Churchill Hospital, Oxford, UK
  5. 5Women's Health Academic Centre, King's College London, Guy's Hospital, London, UK
  6. 6NIHR Biomedical Research Centre at Guy's and St Thomas’ NHS Foundation Trust and King's College London, Guy's Hospital, London, UK
  7. 7Guy's and St Thomas’ NHS Foundation Trust, Guy's Hospital, London, UK
  8. 8Policy Institute, King's College London, London, UK
  9. 9MRC Centre for Transplantation, King's College London, Guys’ Hospital, London, UK
  10. 10Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, UK
  11. 11King's Health Partners, Guy's Hospital, London, UK
  1. Correspondence to Professor Alastair M Buchan; alastair.buchan{at}medsci.ox.ac.uk

Abstract

Introduction Translational research organisations (TROs) are a core component of the UK's expanding research base. Equity of career opportunity is key to ensuring a diverse and internationally competitive workforce. The UK now requires TROs to demonstrate how they are supporting gender equity. Yet, the evidence base for documenting such efforts is sparse. This study is designed to inform the acceleration of women's advancement and leadership in two of the UK's leading TROs—the National Institute for Health Research (NIHR) Biomedical Research Centres (BRCs) in Oxford and London—through the development, application and dissemination of a conceptual framework and measurement tool.

Methods and analysis A cross-sectional retrospective evaluation. A conceptual framework with markers of achievement and corresponding candidate metrics has been specifically designed for this study based on an adapted balanced scorecard approach. It will be refined with an online stakeholder consultation and semistructured interviews to test the face validity and explore practices and mechanisms that influence gender equity in the given settings. Data will be collected via the relevant administrative databases. A comparison of two funding periods (2007–2012 and 2012–2017) will be carried out.

Ethics and dissemination The University of Oxford Clinical Trials and Research Governance Team and the Research and Development Governance Team of Guy's and St Thomas’ National Health Service (NHS) Foundation Trust reviewed the study and deemed it exempt from full ethics review. The results of the study will be used to inform prospective planning and monitoring within the participating NIHR BRCs with a view to accelerating women's advancement and leadership. Both the results of the study and its methodology will be further disseminated to academics and practitioners through the networks of collaborating TROs, relevant conferences and articles in peer-reviewed journals.

  • Gender equity
  • Translational research
  • NIHR Biomedical Research Centre (BRC)
  • Evaluation
  • Research impact assessment

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

https://doi.org/10.1136/bmjopen-2015-009022

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • The study addresses the previously neglected need to assess and monitor gender equity in translational research organisations with a view to accelerating women's advancement and leadership.

    • We anticipate that an adapted balanced scorecard approach will enable clarification and translation into operational terms of the organisation's vision and strategy regarding gender equity.

    • The reliability and validity of this cross-sectional retrospective study will depend on the completeness and accuracy of historical data sets and the practicalities of data extraction from these.

    • If this approach proves feasible and robust in a two-centre study, other centres will be encouraged to apply the same methodology to generate comparative data.

    Introduction

    Translational research organisations (TROs) are important elements of the UK's expanding research infrastructure, with a remit to translate biomedical discoveries into effective therapies for patients. Whereas other industries have long recognised that ‘winning the talent war for women’ is key to their growth,1 the lack of gender equity in academic medicine remains a serious threat to the quality and international competitiveness of translational research. Underutilisation of women's talent and potential in biomedical research, especially at senior levels and in leadership roles, has been well documented.2–4 This deprives biomedical research of women's perspectives5 ,6 and more collaborative leadership styles.7 Moreover, some fields (notably women's and paediatric health) are less likely to be investigated by men than women.8 A relative dearth of women mentors and role models in senior positions not only slows down the advancement of the current generation of women translational researchers, but also presents a major problem for the education and training of the next generation. Finally, gender inequity may be the manifestation of discriminatory practices and unconscious biases,2 for which there should be no place in today's science and society. Participation of both genders fairly and without bias is imperative for the legitimisation of and public support for science, including allocation of public resources.9 For all these reasons, TROs need to demonstrably accelerate women's advancement and leadership.

    In England, the National Institute for Health Research (NIHR) has challenged NIHR-funded TROs to improve gender equity and career advancement for women in biomedical research.10 For example, NIHR Biomedical Research Centres and Units (BRCs and BRUs) are not expected to be short listed and therefore eligible for future funding ‘where the academic partner (generally the Medical School/Faculty of Medicine) has not achieved at least the Silver Award of the Athena SWAN Charter for Women in Science’.11 The Charter's awards recognise different levels of commitment to advancing women’s careers in science, technology, engineering, maths and medicine (STEMM) based on 10 principles.12

    While Athena SWAN awards are useful markers of achievement for higher education institutions and research institutes, they alone are likely to be insufficient for assessing and monitoring the progress of NIHR TROs towards gender equity—not least because they were not designed for such a purpose. We believe that NIHR TROs need to make their own measurable contribution to accelerating women's advancement and leadership in translational research. Our study addresses this previously neglected need by focusing specifically on two of the UK's leading TROs—the NIHR BRC at the Oxford University Hospitals National Health Service (NHS) Foundation Trust and the University of Oxford and the NIHR BRC at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. This approach is likely to be relevant and transferable to other TROs in the UK and (with adaptation) comparable organisations around the world.

    Methods and analysis

    Study overview

    This study has two components. The first is the development of a conceptual framework to assess and monitor gender equity in the participating NIHR BRCs. This paper describes the protocol for the development of the conceptual framework, which will be refined with an online stakeholder consultation and semistructured interviews to test the face validity and explore practices and mechanisms that influence gender equity in the given settings. The second component is the application of the conceptual framework in repeated cross-sectional evaluations resulting in a historical picture of how things have changed over time. Devising and applying a framework will ensure that this process is done consistently on each occasion and provide comparable observational data. It will also provide an indication of the quality and consistency of data over the study time period and the feasibility and resource requirements for obtaining and analysing these data.

    Study aim and objectives

    The aim of the study is to inform the acceleration of women's advancement and leadership in translational research through the development, application and dissemination of a conceptual framework and measurement tool. The objectives of the study are as follows:

    • To devise a conceptual framework which captures the major dimensions of gender equity in the NIHR BRCs and complements the existing forms of performance monitoring and research impact assessment.

    • To assess data using this framework from the administrative databases for the two previous funding rounds (2007–2012 and 2012–2017) within the NIHR BRCs.

    • To create an objective baseline and evidence base that informs prospective planning and monitoring of gender equity.

    Study setting

    The study will be conducted at two NIHR BRCs—partnerships between the UK's leading NHS organisations and universities. Together with the other NIHR-funded TROs, such as BRUs and Collaborations for Leadership in Applied Health Research and Care (CLAHRCs), NIHR BRCs are part of the UK Government's initiative to enable a more effective translation of basic science discoveries to clinically testable interventions, especially drugs and devices.13

    The NIHR BRC at the Oxford University Hospitals NHS Foundation Trust and the University of Oxford was established in 2007 through a competitive NIHR award of £57 million over 5 years, and in 2012 it was awarded a further 5 years funding of £96 million. This funding enables NHS consultants and university academics to devote time and resources to concentrating on translational research across a number of themes. Currently, there are nine ‘vertical’ research themes that have a disease or therapeutic focus (eg, cardiovascular, diabetes, vaccines); five ‘cross-cutting’ research themes that bring together platform technologies (eg, genomic medicine, biomedical informatics and technology, surgical innovation and evaluation); and seven working groups that are set up to develop strategic priorities (eg, cognitive health, molecular diagnostic, and research education and training; http://oxfordbrc.nihr.ac.uk).

    The NIHR BRC at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London was also established in 2007, being awarded £51 million. It successfully bid for a further £68 million in 2012 to fund its work until 2017. This NIHR BRC focuses on taking advances in basic science out of the laboratory and into clinical settings in order to benefit patients at the earliest opportunity, and creates an active partnership between clinical and academic staff. The eight research themes cross-cut with four clusters, which focus on different stages of translational science (http://www.guysandstthomasbrc.nihr.ac.uk).

    Study population

    Unlike Athena SWAN, which focuses on university academic and research staff, our study population is intentionally broader and includes all NHS consultants and university clinical academics funded by the NIHR BRCs (ie, NIHR investigators and NIHR senior investigators); administrative and support staff (ie, NIHR associates); academic and clinical trainees (ie, NIHR trainees); and leaders (including the most senior executive and non-executive committees in both NIHR BRCs). These professional groups have been selected because of their involvement in translational research as well as its administration, support and leadership. Their numbers are also sufficiently high to warrant a retrospective evaluation and prospective planning.

    Conceptual framework development

    Although a comprehensive literature search returned no directly relevant instruments for TROs, we identified the US National Initiative on Gender, Culture and Leadership in Medicine (C-Change) at Brandeis University (http://cchange.brandeis.edu) as an example of current best practice, and established collaboration with the authors of the C-Change Markers of Achievement Index (MAI). This instrument was designed and used in five US medical schools to track temporal patterns indicating progress in leadership and achievement for female faculty and of faculty from racial and ethnic groups under-represented in US medicine.14 One of the authors (LHP), who heads C-Change, shared the C-Change MAI instrument used in US medical schools during the 2014 Mid-Term Review of the NIHR Oxford BRC.

    We developed our conceptual framework (figure 1) by adapting the C-Change MAI instrument and a wider relevant literature using the balanced scorecard approach proposed by Kaplan and Norton.15 The advantage of the balanced scorecard, widely used in the commercial sector and in US healthcare organisations, is its departure from the traditional performance assessment based solely on financial measures, to include intangible assets and capabilities needed for future growth.15 The ‘scorecard’ comprises two backward-looking measures of what has already been achieved (in the original model, value for customers and financial profit) and two forward-looking measures of process (effective and efficient work practices, and learning and growth). This approach enables organisations to clarify their vision and strategy and then translate these into operational terms.16 It has previously been adapted for performance assessment in TROs by Pozen and Kline,17 in academic clinical departments by Rimar and Garstka,18 and in a national health system by El Turabi et al.19 We draw on their work in adapting the balanced scorecard to look at gender equity as an important area of performance improvement and research impact assessment in TROs.

    Figure 1
    Figure 1

    Multidimensional conceptual framework for gender equity assessment and monitoring. IP, intellectual property.

    In keeping with the balanced scorecard approach,15–17 our conceptual framework distinguishes between several levels of performance assessment, planning and monitoring:

    • Performance dimensions represent which domains TROs need to address to promote gender equity in a balanced way;

    • Markers of achievement are specific measures of these performance dimensions;

    • Metrics translate markers of achievement into operational terms and highlight what is important to a given TRO;

    • Targets and milestones express the TRO's goals and strategic objectives, and help monitor progress in achieving them.

    The effectiveness of the conceptual framework will be determined according to the SMART criteria (Specific, Measurable, Assignable, Realistic, Time-related)20 and in compliance with the UK Government's FABRIC framework for performance information systems (Focused, Appropriate, Balanced, Robust, Integrated, Cost-Effective).21 Below we explain the first three levels of the conceptual framework (table 1). Targets and milestones pertaining to the fourth level of the conceptual framework will be established as part of the actual planning process during the next funding period, which is beyond the scope of the current study.

    View this table:
    Table 1

    Markers of achievement and candidate metrics for gender equity assessment and monitoring

    Leadership

    Leadership is key to a TRO's ability to identify research questions and develop research projects that are of great importance to society and can have high translational impact. There are many styles of leadership and diversity of perspectives, which more women in senior positions can make available.6 ,22 For example, studies show that women are more likely than men to exhibit transformative leadership styles, which employ more collaborative and less hierarchical approaches.7 Although differences between male and female leaders may be minimal, the characteristics of leadership style on which women exceeded men are positively associated with leaders’ effectiveness,7 with higher performance and better outcomes in healthcare,23–25 and with the collaborative and team culture preferred by physician scientists in a given setting.26 ,27 Additionally, leaders serve as mentors and role models for the next generation of leaders—so more female leaders in TROs today may mean more transformative leadership styles used by both male and female leaders of tomorrow.28 Even though studies show gender equity at medical school entry and equal leadership aspirations in academic medicine among men and women faculty, the under-representation of the latter in academic medicine leadership roles persists.2 ,29–31

    Markers of achievement for leadership concern women leaders in both non-executive and executive roles. Candidate quantitative metrics include both the absolute and relative numbers of women leaders in post, women promoted to leadership roles from within the organisation, and women leaders recruited from the outside of the organisation. Other candidate metrics include the availability and effectiveness of policies and programmes aimed at gender-sensitive leadership development, retention and succession planning. Candidate metrics may also include external leadership roles and esteem indicators such as membership of professional and scientific organisations, panels and societies.

    Talent

    Talent, by which we mean individuals’ experience and capability in basic and/or translational science, is the cornerstone of a TRO's ability to translate biomedical discoveries into benefits for patients in a creative, cost-effective and timely manner. Studies show that many women are attracted to academic medicine because of their interest in clinical work with patients.32 Therefore, an increase in the number of women researchers is likely to have a positive influence on the clinical focus of biomedical research. Likewise, women in academic medicine appear to be interested in teaching33–35 and mentoring36 more than men, though these differences may not be due to innate traits. An increase in the number of women is therefore likely to enhance the ability of a given TRO to train and mentor the next generation of translational researchers. However, clinical work and teaching have a lower status than research, so engaging with them can impede women's advancement and leadership in TROs. Studies also show that female faculty in academic medicine have less alignment of their personal values with undesirable behaviours often observed in academic medicine, for example, devaluing of social and clinical missions of academic medicine, questionable ethical behaviour, and the necessity for self-promotion to achieve success.31 ,37 ,38 This may suggest that having more women faculty would result in a more values-based approach to research. Research beyond medicine shows the importance of having a critical mass of women as a predictor of their acceptance and success.9 In organisations (outside medicine), where women make up fewer than 15% of the workforce, women are less likely to be accepted in the organisation and less likely to progress their careers.39

    Markers of achievement for talent concern women in several different categories, including investigators and senior investigators who are directly involved in undertaking research; associates who support and administer research led by others; and trainees who undertake research training and career development.40 Candidate quantitative metrics for faculty members include both the absolute and relative numbers of women in different categories. Another important candidate metric is the proportion of women investigators who are substantively employed as NHS consultants and university academics. Candidate qualitative metrics include organisational efforts aimed at the recruitment and retention of women as well as opportunities for women to advance their careers within the organisation.

    Funding

    Funding supports a given TRO to sustain all aspects of its activities, ranging from resourcing research projects, buying out time of NHS consultants to conduct research, providing research training and administering research. Research from both US and European academic medicine suggests that women are awarded fewer grants and of lesser value compared with men.41–45 This lack of parity in funding may indicate not only overt gender differences in the levels of productivity and in certain research areas, but also more covert aspects at play such as gender discrimination46–49 and unconscious bias.50 ,51 Therefore, greater parity in the allocation of funding between men and women will give women more equitable opportunities to conduct research as well as indicate progress towards freedom from gender discrimination and bias.

    Markers of achievement for funding concern women in receipt of different streams of BRC funding, including the main stream of funding that goes towards individual research projects, but also a stream of funding towards buying out NHS consultants’ time (programmed activities) to conduct research, and a stream of funding towards research training awards for academic fellows and research nurses. Candidate quantitative metrics for funding include both the absolute and relative numbers of women in receipt of different streams of BRC funding, average and total amounts of funding received by gender, as well as funding application success rates.

    Outputs

    Outputs express the research-based contribution of a given TRO to the health of local and global patient communities and to the nation's economic and social development. Although there is substantial evidence to suggest that women publish less than men,52–57 there is also evidence to suggest that women publish as much as men,58–64 particularly after adjusting for age and rank.49 ,65 Moreover, women's publications rates are increasing57 ,66 ,67 and can actually exceed men's publication rates in the latter stages of careers.62 Therefore, an increase in the number of research outputs by women will indicate an improved institutional capacity to utilise women's talent in translational research and to support women at different stages of their careers.

    Markers of achievement for research outputs concern all major types of research outputs by women faculty, including not only publications, but also intellectual property and external grants and contracts. Candidate quantitative metrics for publications include both absolute and relative numbers of publications by women according to the type of contribution (first/senior author and any contribution) and the type of publication (high-impact factor journal and any journal) as well as the number of times publications by women and men have been cited. Candidate metrics for intellectual property include both absolute and relative numbers of patents applied for and granted, number of spin-out companies established, and overall income from intellectual property. Candidate metrics for external grants and contracts include the absolute and relative numbers of grants and amount of funding awarded to women.

    Face validity

    In order to ensure that all the relevant elements of the proposed conceptual framework and markers of achievement are identified and included in the study, their face validity will be established. The latter denotes the degree to which the contents of the test and its items, which in our case is the conceptual framework with markers of achievement and metrics, are viewed by test respondents as relevant to the context in which the test is being conducted.68 Therefore, we will consult a panel of stakeholders representing the entire study population and then adjust the conceptual framework and its elements according to their feedback.

    The consultation will be carried out using an anonymous online survey and face-to-face semistructured interviews. The survey instrument will comprise both closed-ended quantitative and open-ended qualitative questions. Respondents will be asked to critically appraise the proposed conceptual framework with markers of achievement and metrics (table 1) as well as elaborate on practices and mechanisms that influence gender equity in the given settings. The survey instrument will also include demographic information, including leadership and staff group, substantive employment, gender, and age. Interviewees will be purposively selected to achieve a representative sample of the study population. An adequate interview sample size will be based on the number of interviews necessary to achieve a saturation point, that is, when additional interviews provide no new themes or categories.

    Data collection

    Data will be collected from the relevant administrative databases within the participating NIHR BRCs in two phases. Phase I for the funding period 2007–2012 will start in Q1 2016 with a consultation with a panel of NIHR investigators, NIHR associates and leaders to establish the face validity of the proposed conceptual framework with markers of achievement and metrics. Phase II for the current funding period (2012–2017) will start in Q2 2016 to coincide with the beginning of planning for the next BRC 5-year funding application (2017–2022).

    Data analysis

    In order to carry out a retrospective evaluation of the two funding periods, yearly data collected from the relevant databases will be aggregated for each of the two 5-year funding periods and analysed statistically within Excel and SPSS. Individual variables will be based on the metrics identified in the framework (figure 1). Descriptive statistics will be used to present data pertaining to markers of achievements in each of the funding periods. Tests of significance will be used to carry out comparisons between the current and previous funding periods and to detect gender differences within a given funding period. Missing data are a common occurrence with observational data; however, at present the nature of any missing data is unknown, that is, whether it is random or non-random. Once this is established, the appropriate method to address this will be applied.69

    In order to aid priority setting and quantification of markers of achievement during the next funding period, comparisons between the two funding periods will be used. To aid planning further, comparisons between the participating NIHR BRCs and their founding academic and clinical partners will be made. Such comparisons are important because women's advancement and leadership within both NIHR BRCs are enabled and constrained by the pool of qualified women within the NHS Trusts and the Universities. For example, the extent to which the number of women investigators within each NIHR BRC can be increased will depend on the number of qualified women consultants and clinical academics in the NHS Trust and the University, respectively.

    Ethics and dissemination

    The University of Oxford Clinical Trials and Research Governance Team and the Research and Development Governance Team of the Guy’s and St Thomas’ NHS Foundation Trust reviewed the study and deemed it exempt from full ethics review on the grounds that it falls outside of the Governance Arrangements for Research Ethics Committees (GAfREC), which stipulate which research studies are required to have ethics review. Once the survey instrument is finalised, it will be assessed against the standards set out in the Central University Research Ethics Committee (CUREC) Checklist 1, and if necessary submitted to the CUREC for review. All data collected from the relevant administrative databases will be held and analysed in compliance with the requirements of the UK Data Protection Act 1998 and other relevant legislation and professional guidance. During analyses, data will be aggregated and anonymised. The results of the study will be published internally within both BRCs and used to inform planning with a view to accelerating women's advancement and leadership. Both the results of the study and its methodology will be further disseminated to academics and practitioners through the networks of collaborating TROs, relevant academic and clinical forums, conferences, and articles in peer-reviewed journals.

    Discussion

    Rationale

    The fundamental rationale for this study stems from the desire of the participating NIHR BRCs to make their own measurable contribution to accelerating women's advancement and leadership in translational research. We extend previous work on performance assessment in translational research,17–19 ,70–72 by focusing on gender equity. Our intention is to develop a new multidimensional conceptual framework for gender equity performance assessment in order to use it both for retrospective evaluation and prospective planning and monitoring with a view to accelerating women's advancement and leadership. We believe that measuring the impact of TROs on gender equity should become an important dimension of their research impact assessment.

    Strengths and limitations

    This study has several strengths and limitations. First, it addresses the previously neglected need to assess and monitor gender equity in TROs, focusing specifically on two of the UK's leading TROs. Although there is the need to accelerate women's advancement and leadership in TROs, empirical studies addressing this need remain scarce, especially in the UK. To the best of our knowledge, this is the first study to empirically investigate gender equity in NIHR BRCs in the UK.

    Second, an adapted balanced scorecard approach enables clarification and translation into action of the organisation's vision and strategy regarding gender equity. This will enable the participating NIHR BRCs to identify areas where improvements are needed and inform planning during the next funding period. In doing so, the balanced scorecard approach will help NIHR BRC leaders to operationalise the organisation's ambitions for accelerating women's advancement and leadership as well as establishing targets and milestones for monitoring progress against the goals and strategic objectives.

    Third, the inclusion in the online stakeholder consultation and semistructured interviews of the questions regarding practices and mechanisms that influence gender equity in the given settings will help identify potential strategies to accelerate women's advancement and leadership. The current evidence base for such strategies is predominantly based on observational studies from North America. Our study will help identify interventions that are most relevant to UK TROs and will propose a rigorous tool to measure their efficacy as part of prospective organisational development and change.

    Fourth, the reliability and validity of this cross-sectional retrospective study will depend on the completeness and accuracy of historical data sets and the practicalities of data extraction from these. The study will rely on data extraction from data sets across two different types of institutions, a university and a healthcare provider organisation. Their information systems use different definitions and data codes, and were not specifically designed for assessing and monitoring gender equity. These may limit the completeness and accuracy of the aggregated data.

    Finally, if this approach proves feasible and robust in a two-centre study, other centres will be encouraged to apply the same methodology to generate comparative data. NIHR BRUs and NIHR CLAHRCs in the UK as well as similar organisations around the world could use the results of the study in order to benchmark their own organisations against the two NIHR BRCs. This may facilitate organisational learning between different TROs and lead to further research seeking to determine comparatively the most effective strategies to accelerate women's advancement and leadership.

    Implications and conclusions

    This study defines and tests a new tool for assessing gender equity in two of the UK's leading NIHR BRCs. The results of the study will inform strategic planning and monitoring during the next funding period with a view to accelerating women's advancement and leadership in the participating NIHR BRCs. In doing so, the study will develop new processes and information systems for the collection and analysis of data on gender equity. These processes and information systems will be refined further through continuous feedback from strategic planning and decision-making. The study will also have wider implications. If the methodology, processes and information systems developed as part of the study prove effective, they can be applied to the other neglected dimensions of equity in translational research such as race and ethnicity, disability, sexual orientation, and age.

    Acknowledgments

    The authors acknowledge valuable suggestions from Paula Adam, Lluís Rovira, Remco Coppen, Alex Pollitt and other participants of the International School on Research Impact Assessment (ISRIA)—Rathenau Instituut Workshop, The Hague, 16 April 2015. They also thank Edward Krupat and David Kern, members of the C-Change research team who helped develop the C-Change Markers of Achievement Index, and to the Josiah Macy, Jr Foundation that funded this prior work. They are particularly grateful to Leslie McMurtry and Helena Mattingley for help with data acquisition.

    References

    1. ↵
      1. McCracken DM
      . Winning the talent war for women. Sometimes it takes a revolution. Harv Bus Rev 2000;78:159–60, 62, 64–7.
      OpenUrlPubMedWeb of Science
    2. ↵
      1. Penny M,
      2. Jeffries R,
      3. Grant J, et al
      . Women and academic medicine: a review of the evidence on female representation. J R Soc Med 2014;107:259–63. doi:10.1177/0141076814528893
      OpenUrlFREE Full Text
    3. ↵
      Catalan Health System Observatory Results Centre. Research in health sciences: data from 2012. Barcelona: Catalan Agency of Health Quality and Assessment, Ministry of Health, Government of Catalonia, 2014.
    4. ↵
      CERCA Institute. Equal opportunities and diversity management plan. Barcelona: CERCA Institute, 2014.
    5. ↵
      1. Pololi LH,
      2. Jones SJ
      . Women faculty: an analysis of their experiences in academic medicine and their coping strategies. Gend Med 2010;7:438–50. doi:10.1016/j.genm.2010.09.006
      OpenUrlCrossRefPubMed
    6. ↵
      1. Shah DN,
      2. Volpe NJ,
      3. Abbuhl SB, et al
      . Gender characteristics among academic ophthalmology leadership, faculty, and residents: results from a cross-sectional survey. Ophthalmic Epidemiol 2010;17:1–6. doi:10.3109/09286580903324892
      OpenUrlPubMed
    7. ↵
      1. Eagly AH,
      2. Johannesen-Schmidt MC,
      3. van Engen ML
      . Transformational, transactional, and laissez-faire leadership styles: a meta-analysis comparing women and men. Psychol Bull 2003;129:569–91. doi:10.1037/0033-2909.129.4.569
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Carnes M,
      2. VandenBosche G,
      3. Agatisa PK, et al
      . Using women's health research to develop women leaders in academic health sciences: the National Centers of Excellence in Women's Health. J Womens Health Gend Based Med 2001;10:39–47. doi:10.1089/152460901750067106
      OpenUrlCrossRefPubMed
    9. ↵
      1. Etzkowitz H,
      2. Kemelgor C,
      3. Neuschatz M, et al
      . The paradox of critical mass for women in science. Science 1994;266:51–4. doi:10.1126/science.7939644
      OpenUrlFREE Full Text
    10. ↵
      1. Donald A,
      2. Harvey PH,
      3. McLean AR
      . Athena SWAN awards: bridging the gender gap in UK science. Nature 2011;478:36. doi:10.1038/478036b
      OpenUrlPubMed
    11. ↵
    12. ↵
      Equality Challenge Unit. About ECU's Athena SWAN Charter. http://www.ecu.ac.uk/equality-charters/athena-swan/about-athena-swan/
    13. ↵
      Department of Health. Best research for best health: a new national health research strategy. London: Department of Health, 2006.
    14. ↵
      1. Krupat E,
      2. Pololi L,
      3. Schnell ER, et al
      . Changing the culture of academic medicine: the C-Change learning action network and its impact at participating medical schools. Acad Med 2013;88:1252–8. doi:10.1097/ACM.0b013e31829e84e0
      OpenUrl
    15. ↵
      1. Kaplan RS,
      2. Norton DP
      . Using the balanced scorecard as a strategic management system. Harv Bus Rev 2005;84:100–9, 150.
      OpenUrl
    16. ↵
      Balanced Scorecard Institute. Balanced Scorecard Basics. http://balancedscorecard.org/Resources/About-the-Balanced-Scorecard
    17. ↵
      1. Pozen R,
      2. Kline H
      . Defining success for translational research organizations. Sci Transl Med 2011;3:94cm20. doi:10.1126/scitranslmed.3002085
      OpenUrlFREE Full Text
    18. ↵
      1. Rimar S,
      2. Garstka SJ
      . The “Balanced Scorecard”: development and implementation in an academic clinical department. Acad Med 1999;74:114–22. doi:10.1097/00001888-199902000-00010
      OpenUrlPubMedWeb of Science
    19. ↵
      1. El Turabi A,
      2. Hallsworth M,
      3. Ling T, et al
      . A novel performance monitoring framework for health research systems: experiences of the National Institute for Health Research in England. Health Res Policy Syst 2011;9:13. doi:10.1186/1478-4505-9-13
      OpenUrlPubMed
    20. ↵
      1. Doran GT
      . There's a S.M.A.R.T. way to write management's goals and objectives. Manag Rev 1981;70:35–6.
      OpenUrl
    21. ↵
      1. HM Treasury, Cabinet Office, National Audit Office,
      2. et al
      . Choosing the right FABRIC: a framework for performance information. London: TSO, 2001.
    22. ↵
      1. Conrad P,
      2. Carr P,
      3. Knight S, et al
      . Hierarchy as a barrier to advancement for women in academic medicine. J Womens Health (Larchmt) 2010;19:799–805. doi:10.1089/jwh.2009.1591
      OpenUrlPubMed
    23. ↵
      1. Meterko M,
      2. Mohr DC,
      3. Young GJ
      . Teamwork culture and patient satisfaction in hospitals. Med Care 2004;42:492–8. doi:10.1097/01.mlr.0000124389.58422.b2
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Hartmann CW,
      2. Meterko M,
      3. Rosen AK, et al
      . Relationship of hospital organizational culture to patient safety climate in the Veterans Health Administration. Med Care Res Rev 2009;66:320–38. doi:10.1177/1077558709331812
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Zazzali JL,
      2. Alexander JA,
      3. Shortell SM, et al
      . Organizational culture and physician satisfaction with dimensions of group practice. Health Serv Res 2007;42:1150–76. doi:10.1111/j.1475-6773.2006.00648.x
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Ovseiko PV,
      2. Buchan AM
      . Organizational culture in an academic health center: an exploratory study using a Competing Values Framework. Acad Med 2012;87:709–18. doi:10.1097/ACM.0b013e3182537983
      OpenUrlPubMed
    27. ↵
      1. Ovseiko PV,
      2. Melham K,
      3. Fowler J, et al
      . Organisational culture and post-merger integration in an academic health centre: a mixed-methods study. BMC Health Serv Res 2015;15:25. doi:10.1186/s12913-014-0673-3
      OpenUrl
    28. ↵
      1. Sambunjak D,
      2. Straus SE,
      3. Marusic A
      . Mentoring in academic medicine: a systematic review. JAMA 2006;296:1103–15. doi:10.1001/jama.296.9.1103
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Richman RC,
      2. Morahan PS,
      3. Cohen DW, et al
      . Advancing women and closing the leadership gap: the Executive Leadership in Academic Medicine (ELAM) program experience. J Womens Health Gend Based Med 2001;10:271–7. doi:10.1089/152460901300140022
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Carnes M,
      2. Devine PG,
      3. Baier Manwell L, et al
      . The effect of an intervention to break the gender bias habit for faculty at one institution: a cluster randomized, controlled trial. Acad Med 2015;90:221–30. doi:10.1097/ACM.0000000000000552
      OpenUrlCrossRefPubMed
    31. ↵
      1. Pololi LH,
      2. Civian JT,
      3. Brennan RT, et al
      . Experiencing the culture of academic medicine: gender matters, a national study. J Gen Intern Med 2013;28:201–7. doi:10.1007/s11606-012-2207-1
      OpenUrlCrossRefPubMed
    32. ↵
      1. Cydulka RK,
      2. D'Onofrio G,
      3. Schneider S, et al
      . Women in academic emergency medicine. Acad Emerg Med 2000;7:999–1007. doi:10.1111/j.1553-2712.2000.tb02091.x
      OpenUrlPubMedWeb of Science
    33. ↵
      1. Mayer AP,
      2. Blair JE,
      3. Ko MG, et al
      . Gender distribution of U.S. medical school faculty by academic track type. Acad Med 2014;89:312–17. doi:10.1097/ACM.0000000000000089
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Andriole DA,
      2. Jeffe DB
      . The road to an academic medicine career: a national cohort study of male and female US medical graduates. Acad Med 2012;87:1722–33. doi:10.1097/ACM.0b013e318271e57b
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Kaplan SH,
      2. Sullivan LM,
      3. Dukes KA, et al
      . Sex differences in academic advancement. Results of a national study of pediatricians. N Engl J Med 1996;335:1282–9. doi:10.1056/NEJM199610243351706
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Dannels S,
      2. McLaughlin J,
      3. Gleason KA, et al
      . Medical school deans’ perceptions of organizational climate: useful indicators for advancement of women faculty and evaluation of a leadership program's impact. Acad Med 2009;84:67–79. doi:10.1097/ACM.0b013e3181906d37
      OpenUrlCrossRefPubMed
    37. ↵
      1. Pololi L,
      2. Kern DE,
      3. Carr P, et al
      . The culture of academic medicine: faculty perceptions of the lack of alignment between individual and institutional values. J Gen Intern Med 2009;24:1289–95. doi:10.1007/s11606-009-1131-5
      OpenUrlCrossRefPubMed
    38. ↵
      1. Pololi LH
      . Changing the culture of academic medicine: perspectives of women faculty. Hanover, NH: Dartmouth College Press, 2010.
    39. ↵
      1. South SJ,
      2. Bonjean CM,
      3. Markham WT, et al
      . Social structure and intergroup interaction: men and women of the Federal Bureaucracy. Am Sociol Rev 1982;47:587–99. doi:10.2307/2095160
      OpenUrlCrossRefWeb of Science
    40. ↵
      National Institute for Health Research (NIHR). Types of membership. http://www.nihr.ac.uk/our-faculty/types-of-membership.htm
    41. ↵
      1. Eloy JA,
      2. Svider PF,
      3. Kovalerchik O, et al
      . Gender differences in successful NIH grant funding in otolaryngology. Otolaryngol Head Neck Surg 2013;149:77–83. doi:10.1177/0194599813486083
      OpenUrlAbstract/FREE Full Text
    42. ↵
      1. Jagsi R,
      2. Motomura AR,
      3. Griffith KA, et al
      . Sex differences in attainment of independent funding by career development awardees. Ann Intern Med 2009;151:804–11. doi:10.7326/0003-4819-151-11-200912010-00009
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Jagsi R,
      2. DeCastro R,
      3. Griffith KA, et al
      . Similarities and differences in the career trajectories of male and female career development award recipients. Acad Med 2011;86:1415–21. doi:10.1097/ACM.0b013e3182305aa6
      OpenUrlCrossRefPubMedWeb of Science
    44. ↵
      1. Head MG,
      2. Fitchett JR,
      3. Cooke MK, et al
      . Differences in research funding for women scientists: a systematic comparison of UK investments in global infectious disease research during 1997–2010. BMJ Open 2013;3:e003362. doi:10.1136/bmjopen-2013-003362
      OpenUrlCrossRefPubMed
    45. ↵
      1. Pohlhaus JR,
      2. Jiang H,
      3. Wagner RM, et al
      . Sex differences in application, success, and funding rates for NIH extramural programs. Acad Med 2011;86:759–67. doi:10.1097/ACM.0b013e31821836ff
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      1. Tacconelli E,
      2. Poljak M,
      3. Cacace M, et al
      . Science without meritocracy. Discrimination among European specialists in infectious diseases and clinical microbiology: a questionnaire survey. BMJ Open 2012;2:pii: e001993. doi:10.1136/bmjopen-2012-001993
      OpenUrl
    47. ↵
      1. Tosi LL,
      2. Mankin HJ
      . Ensuring the success of women in academic orthopaedics. Clin Orthop Relat Res 1998;(356):254–63. doi:10.1097/00003086-199811000-00034
    48. ↵
      1. Kass RB,
      2. Souba WW,
      3. Thorndyke LE
      . Challenges confronting female surgical leaders: overcoming the barriers. J Surg Res 2006;132:179–87. doi:10.1016/j.jss.2006.02.009
      OpenUrlCrossRefPubMedWeb of Science
    49. ↵
      1. Carr PL,
      2. Friedman RH,
      3. Moskowitz MA, et al
      . Comparing the status of women and men in academic medicine. Ann Intern Med 1993;119:908–13. doi:10.7326/0003-4819-119-9-199311010-00008
      OpenUrlCrossRefPubMedWeb of Science
    50. ↵
      1. Carr PL,
      2. Szalacha L,
      3. Barnett R, et al
      . A “ton of feathers”: gender discrimination in academic medical careers and how to manage it. J Womens Health (Larchmt) 2003;12:1009–18. doi:10.1089/154099903322643938
      OpenUrlPubMed
    51. ↵
      1. Abbuhl S,
      2. Bristol MN,
      3. Ashfaq H, et al
      . Examining faculty awards for gender equity and evolving values. J Gen Intern Med 2010;25:57–60. doi:10.1007/s11606-009-1092-8
      OpenUrlPubMed
    52. ↵
      1. Barnett RC,
      2. Carr P,
      3. Boisnier AD, et al
      . Relationships of gender and career motivation to medical faculty members’ production of academic publications. Acad Med 1998;73:180–6. doi:10.1097/00001888-199802000-00017
      OpenUrlCrossRefPubMedWeb of Science
    53. ↵
      1. Steger B,
      2. Colvin HP,
      3. Rieder J
      . Scientific activity and working hours of physicians in university hospitals: results from the Innsbruck and Salzburg physician lifestyle assessment (TISPLA). Wien Klin Wochenschr 2009;121:685–9. doi:10.1007/s00508-009-1226-3
      OpenUrlPubMed
    54. ↵
      1. Eloy JA,
      2. Svider P,
      3. Chandrasekhar SS, et al
      . Gender disparities in scholarly productivity within academic otolaryngology departments. Otolaryngol Head Neck Surg 2013;148:215–22. doi:10.1177/0194599812466055
      OpenUrlAbstract/FREE Full Text
    55. ↵
      1. Morton MJ,
      2. Sonnad SS
      . Women on professional society and journal editorial boards. J Natl Med Assoc 2007;99:764–71.
      OpenUrlPubMed
    56. ↵
      1. Pashkova AA,
      2. Svider PF,
      3. Chang CY, et al
      . Gender disparity among US anaesthesiologists: are women underrepresented in academic ranks and scholarly productivity? Acta Anaesth Scand 2013;57:1058–64. doi:10.1111/aas.12141
      OpenUrlCrossRefPubMed
    57. ↵
      1. Sidhu R,
      2. Rajashekhar P,
      3. Lavin VL, et al
      . The gender imbalance in academic medicine: a study of female authorship in the United Kingdom. J R Soc Med 2009;102:337–42. doi:10.1258/jrsm.2009.080378
      OpenUrlAbstract/FREE Full Text
    58. ↵
      1. Tinjum BE,
      2. Getto L,
      3. Tiedemann J, et al
      . Female authorship in emergency medicine parallels women practicing academic emergency medicine. J Emerg Med 2011;41:723–7. doi:10.1016/j.jemermed.2010.04.014
      OpenUrlPubMed
    59. ↵
      1. Bland CJ,
      2. Center BA,
      3. Finstad DA, et al
      . A theoretical, practical, predictive model of faculty and department research productivity. Acad Med 2005;80:225–37. doi:10.1097/00001888-200503000-00006
      OpenUrlCrossRefPubMedWeb of Science
    60. ↵
      1. Broaddus VC,
      2. Feigal DW Jr.
      . Starting an academic career. A survey of junior academic pulmonary physicians. Chest 1994;105:1858–63. doi:10.1378/chest.105.6.1858
      OpenUrlCrossRefPubMedWeb of Science
    61. ↵
      1. Housri N,
      2. Cheung MC,
      3. Koniaris LG, et al
      . Scientific impact of women in academic surgery. J Surg Res 2008;148:13–16. doi:10.1016/j.jss.2008.02.015
      OpenUrlCrossRefPubMed
    62. ↵
      1. Reed DA,
      2. Enders F,
      3. Lindor R, et al
      . Gender differences in academic productivity and leadership appointments of physicians throughout academic careers. Acad Med 2011;86:43–7. doi:10.1097/ACM.0b013e3181ff9ff2
      OpenUrlCrossRefPubMed
    63. ↵
      1. Sadeghpour M,
      2. Bernstein I,
      3. Ko C, et al
      . Role of sex in academic dermatology: results from a national survey. Arch Dermatol 2012;148:809–14. doi:10.1001/archdermatol.2011.3617
      OpenUrlPubMed
    64. ↵
      1. Vydareny KH,
      2. Waldrop SM,
      3. Jackson VP, et al
      . Career advancement of men and women in academic radiology: is the playing field level? Acad Radiol 2000;7:493–501. doi:10.1016/S1076-6332(00)80321-9
      OpenUrlCrossRefPubMedWeb of Science
    65. ↵
      1. Wright AL,
      2. Schwindt LA,
      3. Bassford TL, et al
      . Gender differences in academic advancement: patterns, causes, and potential solutions in one US College of Medicine. Acad Med 2003;78:500–8. doi:10.1097/00001888-200305000-00015
      OpenUrlCrossRefPubMedWeb of Science
    66. ↵
      1. Amering M,
      2. Schrank B,
      3. Sibitz I
      . The gender gap in high-impact psychiatry journals. Acad Med 2011;86:946–52. doi:10.1097/ACM.0b013e3182222887
      OpenUrlCrossRefPubMedWeb of Science
    67. ↵
      1. Feramisco JD,
      2. Leitenberger JJ,
      3. Redfern SI, et al
      . A gender gap in the dermatology literature? Cross-sectional analysis of manuscript authorship trends in dermatology journals during 3 decades. J Am Acad Dermatol 2009;60:63–9. doi:10.1016/j.jaad.2008.06.044
      OpenUrlCrossRefPubMedWeb of Science
    68. ↵
      1. Holden RB
      . Face validity. In: Weiner IB, Craighead WE, eds. The Corsini encyclopedia of psychology. 4th ed. Hoboken, NJ: Wiley, 2010:637–8.
    69. ↵
      1. Rubin DB,
      2. Little RJA
      . Statistical analysis with missing data. 2nd edn. New York: Wiley, 2002.
    70. ↵
      1. Rajan A,
      2. Caldas C,
      3. van Luenen H, et al
      . Assessing excellence in translational cancer research: a consensus based framework. J Transl Med 2013;11:274. doi:10.1186/1479-5876-11-274
      OpenUrl
    71. ↵
      1. Dembe AE,
      2. Lynch MS,
      3. Gugiu PC, et al
      . The translational research impact scale: development, construct validity, and reliability testing. Eval Health Prof 2014;37:50–70. doi:10.1177/0163278713506112
      OpenUrlAbstract/FREE Full Text
    72. ↵
      1. Grether M,
      2. Eickelberg O,
      3. Mall MA, et al
      . New metrics for translational research. Lancet Respir Med 2014;2:e13–4. doi:10.1016/S2213-2600(14)70150-X
      OpenUrlCrossRefPubMed

    Footnotes

    • Twitter Follow Trisha Greenhalgh at @trishgreenhalgh

    • Contributors AMB, PVO, VK, LHP, LDE, CW, GML and KMC conceived and designed the study. PVO and LDE drafted the manuscript and integrated critical feedback and important content from all of the other authors. All of the authors read and approved the final version of the manuscript.

    • Funding The research was funded/supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre and by the NIHR Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. LHP was a panel member of the NIHR Oxford BRC Mid-Term Review 2014, and received an honorarium from the NIHR Oxford BRC. LDE was supported by the John Fell Fund and the Vice-Chancellor's Diversity Fund, University of Oxford. KMC and GML hold NIHR Senior Investigator Awards.

    • Disclaimer The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

    • Competing interests KMC is the Director, and VK is the Head of Operations of the NIHR Oxford BRC. AMB was the founding Director of the NIHR Oxford BRC. GML is the Director of the NIHR BRC at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. TG represented Medicine on the Equality and Diversity Panel for the 2014 Research Excellence Framework (REF); in that role, she was involved in assessing universities’ procedures for assuring equality (by gender, ethnicity, disability and sexual orientation) in their decisions on which individuals to submit to the REF.

    • Ethics approval The University of Oxford Clinical Trials and Research Governance Team and the Research and Development Governance Team of the Guy’s and St Thomas’ NHS Foundation Trust reviewed the study and deemed it exempt from full ethics review on the grounds that it falls outside of the Governance Arrangements for Research Ethics Committees (GAfREC), which stipulate which research studies are required to have ethics review.

    • Provenance and peer review Not commissioned; externally peer reviewed.