Caution urged amid wide variation in response rates to friends and family test

The first results of the government’s new NHS friends and family test should be treated with caution because of the huge variation in response rates, commentators have warned.

The test, which asks patients whether they would recommend their hospital ward or accident and emergency department to their friends and family, has been hailed by the government as a “game changer” in giving patients a stronger voice in deciding whether their care is good enough or not—and to encourage hospitals to make improvements.

The first results, published on 30 July, showed that inpatient services were more highly recommended than accident and emergency services, while specialist hospitals tended to have higher scores for inpatient services.1

In June, just 36 hospital wards out of 4500 across the country scored an overall negative figure, down from 66 in April. For accident and emergency in June, just one service received a negative score.

The results were gathered from 400 000 patients in April, May, and June 2013.

But despite much government fanfare, critics have raised concerns about whether the test is fit for purpose,2 while others warned that the test should not be relied upon in isolation, and was open to gaming.

NHS England insisted that the initial results contained some “home truths” for trusts, but admitted that low response rates in some areas could have a “dramatically disproportionate” impact on scores.

While the England-wide response rate for both inpatient and accident and emergency surveys was 13.1%, this masked huge variation. For example, in June, accident and emergency at South Warwickshire NHS Foundation Trust achieved a score of 100, but from a response rate of just 0.1% (three people). In contrast, accident and emergency at the Royal Free London Foundation Trust in London achieved a score of 41, but from a huge response rate of 49.2% (2319 people).

East Kent Hospitals University Foundation Trust and Walsall Healthcare NHS Trust both scored 70 on inpatient ratings in June, but East Kent’s score was based on a response rate of just 2.7% (169 patients), while Walsall had a response rate of 66.1% (1019 patients).

Overall, accident and emergency service scores ranged from 100 to minus 13, with the top 10 trusts rated between 100 and 79. The scores for inpatients ranged from 100 to 43.

The design of the test allows services to be scored between +100 and −100, with scores calculated by analysing responses and categorising them into “promoters” (those who said they were “extremely likely” to recommend the service); “neutral” (those who said “likely”) and “detractors” (those who said they were “neither unlikely nor likely,” “unlikely,” or “extremely unlikely”).

Inpatient data were submitted by all 157 acute NHS trusts in England plus private sector providers, and accident and emergency data by all 144 providers of relevant A&E services.

Tim Kelsey, NHS England’s national director for patients and information, said, “From this first publication, we can see a significant and real variation in the quality of customer service across the NHS. There are home truths here and everyone will expect those trusts who have large numbers of their patients choosing not to recommend their services to respond as quickly as possible.

“It is important, however, this early data is treated carefully. Low response rates can have a dramatically disproportionate impact on scores. As more and more patients respond, the data will become more and more robust.”

England’s health secretary, Jeremy Hunt, described the launch as “a historic moment for the NHS,” adding, “This simple test will throw a spotlight on both good and poor care, and sends a clear signal to every NHS organisation that patients must come first every time.”

Johnny Marshall, director of policy at the NHS Confederation, said that the results were useful, but cautioned that they should not be used in isolation to judge services.

“Given the variation in the way responses are collected from patients, we need to be alive to the impact this will have on results,” he said. “While the feedback from these tests is really useful to add to the wider pool of information we have to judge trusts on the standards of their services, we shouldn’t necessarily use it in isolation to make initial comparisons between organisations.”

Jocelyn Cornwell, director of the Point of Care Foundation, a charity set up in the wake of the scandal at Mid Staffordshire NHS Foundation Trust, said collecting feedback was important, but that the test was a “blunt instrument” that was open to gaming.

She added, “The way in which the data for the friends and family test is collected varies widely and is open to gaming. People who respond are not part of a random sample, but are self selecting or worse, are encouraged to respond by staff. Clearly there is a temptation for staff to encourage responses from patients who they feel will respond positively, especially as a positive result is linked to financial reward.”

NHS England plans to extend the test to maternity services in October 2013, and to GP practices, community services and mental health services by the end of December 2014, with all other NHS services following by the end of March 2015.