Relationship between quality of care and choice of clinical computing system: retrospective analysis of family practice performance under the UK's quality and outcomes framework

Evangelos Kontopantelis; Iain Buchan; David Reeves; Kath Checkland; Tim Doran

doi:10.1136/bmjopen-2013-003190

Responses

This article has a correction. Please see:

Correction - August 01, 2013

PDF

PDF +
Supplementary
Material

XML

General practice / Family practice

Research

Relationship between quality of care and choice of clinical computing system: retrospective analysis of family practice performance under the UK's quality and outcomes framework

Compose a Response to This Article

Compose Response

Title *

Author Information

Contributors
First Name and Middle Initial * First or given name, e.g. 'Peter'. Last Name * Your last, or family, name, e.g. 'MacMoody'. Email Address * Your email address, e.g. higgs-boson@gmail.com Occupation * Your role and/or occupation, e.g. 'Orthopedic Surgeon'. Affiliation * Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.

Statement of Competing Interests

Competing interests? *

Yes

Please describe the competing interests

PLEASE NOTE:

A rapid response is a moderated but not peer reviewed online response to a published article in a BMJ journal; it will not receive a DOI and will not be indexed unless it is also republished as a Letter, Correspondence or as other content. Find out more about rapid responses.
We intend to post all responses which are approved by the Editor, within 14 days (BMJ Journals) or 24 hours (The BMJ), however timeframes cannot be guaranteed. Responses must comply with our requirements and should contribute substantially to the topic, but it is at our absolute discretion whether we publish a response, and we reserve the right to edit or remove responses before and after publication and also republish some or all in other BMJ publications, including third party local editions in other countries and languages
Our requirements are stated in our rapid response terms and conditions and must be read. These include ensuring that: i) you do not include any illustrative content including tables and graphs, ii) you do not include any information that includes specifics about any patients,iii) you do not include any original data, unless it has already been published in a peer reviewed journal and you have included a reference, iv) your response is lawful, not defamatory, original and accurate, v) you declare any competing interests, vi) you understand that your name and other personal details set out in our rapid response terms and conditions will be published with any responses we publish and vii) you understand that once a response is published, we may continue to publish your response and/or edit or remove it in the future.
By submitting this rapid response you are agreeing to our terms and conditions for rapid responses and understand that your personal data will be processed in accordance with those terms and our privacy notice.

I agree to and have read the terms and conditions for rapid responses and BMJ Privacy Notice *

CAPTCHA

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

Other responses

Jump to comment:

Re:Data from HSCIC: if the article needed correction of the underlying data, should the authors review the conclusions reached?
Evan Kontopantelis
Published on: 23 November 2015
Data from HSCIC: if the article needed correction of the underlying data, should the authors review the conclusions reached?
Mary Hawking
Published on: 23 November 2015
Re:Further concerns on methods, results and conclusions
Evangelos Kontopantelis
Published on: 23 November 2015
Further concerns on methods, results and conclusions
Julia Hippisley-Cox
Published on: 23 November 2015
So many concerns, where to begin?
Mary Hawking
Published on: 23 November 2015
Re:So many concerns, where to begin?
Evan Kontopantelis
Published on: 23 November 2015

Published on: 23 November 2015
Re:Data from HSCIC: if the article needed correction of the underlying data, should the authors review the conclusions reached?
- Evan Kontopantelis, Researcher
We appreciate the interest in the paper and the further criticisms.
In the London SHA there were 1,504 practices in 2010/11, not 463 (plus we never reported such a number). 24 of these were using SystmOne, giving it a market share of approximately 1.6%.
We can't find evidence of the data corruption you suggest. As explained before, the correction related to a table label and was purely of a descriptive...
Show More

We appreciate the interest in the paper and the further criticisms.
In the London SHA there were 1,504 practices in 2010/11, not 463 (plus we never reported such a number). 24 of these were using SystmOne, giving it a market share of approximately 1.6%.
We can't find evidence of the data corruption you suggest. As explained before, the correction related to a table label and was purely of a descriptive nature: not affecting the results and, consequently, not affecting the conclusions.
We made every effort to clarify this in previous replies.
Conflict of Interest:
None declared
Show Less

Conflict of Interest:
None declared.
- Back to top
Published on: 23 November 2015
Data from HSCIC: if the article needed correction of the underlying data, should the authors review the conclusions reached?
- Mary Hawking, retired GP
I have now received the tables provided to Manchester University by HSCIC listing GP practice systems by SHA and PCT dated April 2011 and market share dated May 2013.
In the practice GP practice table, there are 463 practices listed in London SHA.
Of these 10 are listed as using TPP SystmOne - 8 in Hounslow and 2 in Hillingdon.
This suggests that the data in the table regarding system distrib...
Show More

I have now received the tables provided to Manchester University by HSCIC listing GP practice systems by SHA and PCT dated April 2011 and market share dated May 2013.
In the practice GP practice table, there are 463 practices listed in London SHA.
Of these 10 are listed as using TPP SystmOne - 8 in Hounslow and 2 in Hillingdon.
This suggests that the data in the table regarding system distribution has somehow become corrupted, and therefore the analysis of any QOF data/quality indicators for practices and effects of systems used must also be open to question:could I suggest that the authors revisit the paper and their conclusions - or at the very least clarify how the links between system use in 2011 were linked to QOF data tables?
When an author admits errors in the article by publishing a correction, it is useful to know whether the errors affect the outcomes and conclusions.
Conflict of Interest:
None declared
Show Less

Conflict of Interest:
None declared.
- Back to top
Published on: 23 November 2015
Re:Further concerns on methods, results and conclusions
- Evangelos Kontopantelis, Researcher
- Other Contributors:
  Tim Doran, David Reeves, Iain Buchan and Kath Checkland
We thank Professor Hippisley-Cox and Dr O'Hanlon for their feedback on our BMJ Open paper (3, e003190). We address each point raised below:
1.The authors have undertaken a cross-sectional study - this is weakest study design and the results of such studies can only ever signal associations rather than attribute causality. Limited information is provided on the 'adjustments' made in the analyses meaning the reade...
Show More

We thank Professor Hippisley-Cox and Dr O'Hanlon for their feedback on our BMJ Open paper (3, e003190). We address each point raised below:
1.The authors have undertaken a cross-sectional study - this is weakest study design and the results of such studies can only ever signal associations rather than attribute causality. Limited information is provided on the 'adjustments' made in the analyses meaning the reader cannot be confident that any of the tiny differences relate to the computer system rather than population characteristics (eg socio- demographic, lifestyle or levels of morbidity) or unmeasured confounding factors.
Response: A) In the limitations section of our paper we acknowledge the difficulties of causal inference in relation to the associations uncovered with this study design. Nevertheless, there is a strong association between clinical information system used and performance as measured by the QOF, which needs to be explained. B) Although the differences between computer systems mean scores we report are relatively small (1.4%-2.7%), as discussed in the paper, the combined effect at the national level is substantial, and is measurable in terms of many thousands of patients. C) We in fact do specify all of the covariates controlled for in the analysis, both in the text and in Table 3, which gives descriptive statistics on each for each type of computer system. We included a good range of both practice and population characteristics, but do acknowledge that we were necessarily limited in the types of covariates available to us. In particular, no patient-level covariates are recorded in QMAS, such as disease severity, obesity, or lifestyle factors, which might impinge on a practice's ability to fully meet the indicators for a patient. We anticipate that to some degree these factors are captured by some of the practice-level covariates, such a percentage of elderly patients, IMD and region, though residual confounding may still be present. And as with all observational studies the possibility of unmeasured confounding factors remains and we agree that further work investigating potential confounders is necessary.
2.The authors have used multiple statistical testing to compare results by computer system. Whilst they have selected a p value of 0.05 (which is arguably too lenient given the number of tests undertaken), they fail to mention what constitutes a clinically/practically important difference. The paper is then written as if they found important differences in outcomes by GP computer system when in reality the differences are negligible (something which was also pointed out by the peer reviewer). There is also an assumption that the QOF measures quality of care which is debatable.
Response: A) Whilst we did use an alpha-level of 5% to designate statistical significance, we did not base our interpretation of the results on significance alone: in view of the large number of practices in the study (8000plus), even very weak associations will reach statistical significance. Instead, we focused on the size of the effects and we have discussed how these effects, although appearing small in absolute terms, can have a large cumulative effect at the population level. B) Whilst we suggest that there may be a link between practice performance on the QOF and quality of care (the purpose of the QOF being to measure and reward quality of care), we do not assume this, and are careful throughout the paper not to conflate the two. We address the validity of using QOF measures as a proxy of quality of care in the limitations section (for example: "we report quality of recorded care and there may be differences with care actually delivered"). We also acknowledge that QOF- related activities represent a fraction of practices' clinical work, but the QOF covers the management of several major chronic conditions, and performance on QOF indicators determines the allocation of over ?1 billion of NHS funding every year. Any association between practice performance on these measures and the use of clinical computing systems therefore warrants investigation.
3.The authors have undertaken a study about GP computer systems and have failed to spot there is no system called ProdSysOneX. This demonstrates a surprising lack of knowledge about General Practice by the researchers and a potential lack of documentation/validation by the HSCIC. It also suggests authors didn't verify the data obtained from the HSCIC on receipt. Simple measures could have prevented this before publication including a google search, visit to the TPP website, email to system suppliers or discussion with any practising GP. Perhaps it should also have been picked up during the peer review process?
Response: The 'ProdSysOneX' title is used in the HSCIC dataset to designate the SystmOne system, and this designation was unfortunately carried over into the paper. Embarrassing as this is, a clinical information system mislabelling is irrelevant to the quality of the dataset, validity of the methods used and the findings. Please note that: i) one of the authors is a practising GP; ii) we did try to contact system vendors to involve them in this research but they did not respond.
4.The errors in table 3 are particularly noticeable as it states that the predominant system in use in London is ProdSysOneX. This is incorrect since EMIS was and is the predominant supplier in London. The authors have tried to correct their mistakes by re-labeling the columns and publishing a revised paper on the Manchester university website. However this still has very basic errors with inconsistencies between the text and the tables: the text of the results states that LV is used by 24.8% of practices in London and 50.5% of practices in the South East whereas the revised table states LV is used in 50.5% of London and 36.8% of the South East. How can the reader be confident that these errors do not affect the multilevel analysis which includes SHA as a level?
Response: This issue has been discussed before and BMJ Open is in the process of issuing a correction. As explained in a previous response, this is due to a mislabelling of the Strategic Health Authorities and only affected the specific table. It is not a data error, thus the analyses are not affected. If you still have reservations about the findings we will be happy to share the data (provided the HSCIC agrees to amend the licence) and you would be welcome to perform your own analyses.
5.The supplementary tables had whole sections where the data appear to be missing - is this intentional?
Response: Yes, these are pairwise comparisons between systems (tables A-3 to A-5), with adjusted p-values for the multiple comparisons.
6.The authors have arrived at a key message "Researchers that utilise primary care databases, which collect data from a single clinical system need to be cautious when generalizing their findings to all English practices". Whilst issues of generalisability of research findings are important, this was not the aim of the paper, the study was not designed to demonstrate this and no results have been presented to support the authors key message.
Response: The message is a logical conclusion from the results of the study. Although the study did not set out to test the generalisability of research based on a single primary care database (PCD), we need to report a major finding in this regard. If choice of clinical system can explain more variation in quality of care as measured by the QOF than area deprivation or other characteristics, that needs to be acknowledged in any analysis that uses data from a single system. We do not feel more results are needed to support that statement. As researchers who use PCDs, we certainly do not want to discredit PCD research; we suggest that PCD based research could be improved by harnessing more heterogeneous sources of data.
7.Many of the higher ranked systems rely on 3rd party add-ons to optimise their QOF figures. More modern systems (for example EMIS Web) have new features designed to improve clinical care and outcomes for patients as well as enable practices to maximise their 'QOF points'.
Response: Thank you for pointing out the QOF related decision support aspects, which may well be important. Unfortunately the uptake of EMIS Web has been insufficient to include it in this study, and we had no data on the use of QOF add-ons at practice level. We appreciate that the results are not ideal for EMIS PCS but the aim of the paper was to identify whether differences in levels of care (as measured by the QOF) exist between systems, and to argue for better integration if we found any. We considered anonymising the products, but because we were reporting market shares alongside performance data felt these would be readily identifiable from that in any case.
Conflict of Interest:
No conflict of interest whatsoever!
Show Less

Conflict of Interest:
None declared.
- Back to top
Published on: 23 November 2015
Further concerns on methods, results and conclusions
- Julia Hippisley-Cox, Professor of General practice & Clinical Epidemiology
- Other Contributors:
  Dr Shaun O'Hanlon, Clinical Director EMIS
Dear Trish
I'm afraid we also have many concerns regarding this paper in addition to those already raised by Mary Hawking.
1. The authors have undertaken a cross-sectional study - this is weakest study design and the results of such studies can only ever signal associations rather than attribute causality. Limited information is provided on the 'adjustments' made in the analyses meaning the reader...
Show More

Dear Trish
I'm afraid we also have many concerns regarding this paper in addition to those already raised by Mary Hawking.
1. The authors have undertaken a cross-sectional study - this is weakest study design and the results of such studies can only ever signal associations rather than attribute causality. Limited information is provided on the 'adjustments' made in the analyses meaning the reader cannot be confident that any of the tiny differences relate to the computer system rather than population characteristics (eg socio- demographic, lifestyle or levels of morbidity) or unmeasured confounding factors.
2. The authors have used multiple statistical testing to compare results by computer system. Whilst they have selected a p value of 0.05 (which is arguably too lenient given the number of tests undertaken), they fail to mention what constitutes a clinically/practically important difference. The paper is then written as if they found important differences in outcomes by GP computer system when in reality the differences are negligible (something which was also pointed out by the peer reviewer). There is also an assumption that the QOF measures quality of care which is debatable.
3. The authors have undertaken a study about GP computer systems and have failed to spot there is no system called ProdSysOneX. This demonstrates a surprising lack of knowledge about General Practice by the researchers and a potential lack of documentation/validation by the HSCIC. It also suggests authors didn't verify the data obtained from the HSCIC on receipt. Simple measures could have prevented this before publication including a google search, visit to the TPP website, email to system suppliers or discussion with any practising GP. Perhaps it should also have been picked up during the peer review process?
4. The errors in table 3 are particularly noticeable as it states that the predominant system in use in London is ProdSysOneX. This is incorrect since EMIS was and is the predominant supplier in London. The authors have tried to correct their mistakes by re-labeling the columns and publishing a revised paper on the Manchester university website. However this still has very basic errors with inconsistencies between the text and the tables: the text of the results states that LV is used by 24.8% of practices in London and 50.5% of practices in the South East whereas the revised table states LV is used in 50.5% of London and 36.8% of the South East. How can the reader be confident that these errors do not affect the multilevel analysis which includes SHA as a level?
5. The supplementary tables had whole sections where the data appear to be missing - is this intentional?
6. The authors have arrived at a key message "Researchers that utilise primary care databases, which collect data from a single clinical system need to be cautious when generalizing their findings to all English practices". Whilst issues of generalisability of research findings are important, this was not the aim of the paper, the study was not designed to demonstrate this and no results have been presented to support the authors key message.
7. Many of the higher ranked systems rely on 3rd party add-ons to optimise their QOF figures. More modern systems (for example EMIS Web) have new features designed to improve clinical care and outcomes for patients as well as enable practices to maximise their 'QOF points'.
Conflict of Interest:
SOH is a director of EMIS - leading commercial supplier for GP computer systems in the UK. Both authors are directors of QResearch (www.qresearch.org) - a research database and joint not-for-profit partnership between EMIS and the University of Nottingham
Show Less

Conflict of Interest:
None declared.
- Back to top
Published on: 23 November 2015
So many concerns, where to begin?
- Mary Hawking, retired GP
There would appear to be a number of inaccuracies in this article which I am very much afraid make the conclusions hard to sustain - and I would urge anyone thinking of using this article to guide their choice of GP IT system at any level to think again!
Just a few points:- The description of GP computing is wrong: GP systems became 50% reimbursable in 1990 - not 1998 - and the reason was to bring them under DH...
Show More

There would appear to be a number of inaccuracies in this article which I am very much afraid make the conclusions hard to sustain - and I would urge anyone thinking of using this article to guide their choice of GP IT system at any level to think again!
Just a few points:- The description of GP computing is wrong: GP systems became 50% reimbursable in 1990 - not 1998 - and the reason was to bring them under DH control: they had to be RFA (Requirement for Accreditation) compliant. Originally QOF had 1050 points, 900 of them in the Clinical Domain: this has now reduced (2013/14) to 900, some of which arare not disease related. At no time were the points as low as 100.
Even more worrying is the data on the GP systems in use (table 3): if TPP SystmOne (assuming that this is the system meant by "TPP by ProdSysOneX") had 58.83% of the practices in London (where it is not the LSP GP system) and only 11.30% in Yorkshire and the Humber (where it is), this had not been previously noticed by those of us with a keen interest in GP IT! (I have sent an FOI to the HSCIC - stated to be the source of the information)
Unfortunately my knowledge of statistical analysis is insufficicient to comment on the conclusions regarding which systems are associated with higher scores in 26 unspecified Clinical Indicators - but it would not appear to me that the data supports the conclusions reached in the article.
Please be careful!
Conflict of Interest:
None declared
Show Less

Conflict of Interest:
None declared.
- Back to top
Published on: 23 November 2015
Re:So many concerns, where to begin?
- Evan Kontopantelis, Researcher
Dear Mary
Thank you for your diligent reading and useful comments. To answer the points you raise:
1) The issue with table 3 has already been identified. Basically the labels for the SHA are wrong. Apologies for this.
2) In the database from the HSCIC SystmOne was labelled as appears on the paper, ProdSysOneX. Embarrassingly, I did not query the name
The BMJ has already been notified a...
Show More

Dear Mary
Thank you for your diligent reading and useful comments. To answer the points you raise:
1) The issue with table 3 has already been identified. Basically the labels for the SHA are wrong. Apologies for this.
2) In the database from the HSCIC SystmOne was labelled as appears on the paper, ProdSysOneX. Embarrassingly, I did not query the name
The BMJ has already been notified about these two issues, which unfortunately were picked up post production, largely thanks to Dr Parry from TPP.
In the meantime you can find the corrected version here: https://www.escholar.manchester.ac.uk/uk-ac-man-scw:201710
However, I can reassure you that these typos do not affect the findings, the conclusions or the message of the paper in the slightest: system seems to be a strong predictor of levels of quality of care, as measured by the QOF.
The other issues you raise:
1) QOF points. We don't really say there are 100 QOF points. The outcome is the percentage of QOF points achieved (which obviously is constrained in the 0-100 range). For a break-down of the points please see the footnotes in table 2
2) Reimbursement in 1998. Our comment is based on this document, dated 8 Dec 1997 and outlining the 1998 reimbursement process: FHSL (97)47: Reimbursement of GP computing costs http://webarchive.nationalarchives.gov.uk/+/www.dh.gov.uk/en/Publicationsandstatistics/Lettersandcirculars/Familyhealthservicesletters/DH_4017754
However, we will happy to make amendments if you can provide us with a reference. In any way, this is again not directly relevant to the message of the paper.
Thanks again, and I hope this clarifies the issues you raise. I will be very happy to discuss further if there's anything else you spot!
All the best
Evan
Show Less

Conflict of Interest:
None declared.
- Back to top

Main menu

This article has a correction. Please see:

Plain text

PLEASE NOTE:

Vertical Tabs

Other responses

Jump to comment:

Conflict of Interest:

Conflict of Interest:

Conflict of Interest:

Conflict of Interest:

Conflict of Interest:

Log in using your username and password

Main menu

Log in using your username and password

You are here

This article has a correction. Please see:

Plain text

PLEASE NOTE:

Vertical Tabs

Other responses

Jump to comment:

Conflict of Interest:

Conflict of Interest:

Conflict of Interest:

Conflict of Interest:

Conflict of Interest:

Read the full text or download the PDF:

Log in using your username and password