Why the FDA can’t protect the public

Getting approval

Representatives from Cyberonics offered no definitive explanation during the FDA meeting of how the device stopped or reduced seizures, but they had three studies, E03, E04, and E05, to show its safety and efficacy.

Two of the studies, E03 and E05, involved 313 patients with treatment resistant partial seizures randomised to high or low dose stimulation. The low stimulation arm was intended to avoid the problem of an unblinded placebo arm because all patients would be implanted and told they were receiving stimulation. The studies did not include a medical treatment arm for comparison, leaving unanswered the question of whether either treatment arm was superior to existing care.

Researchers reported that 25% of patients in the high stimulation arms of the trials achieved the primary end point: a 50% reduction in seizure frequency from baseline. However, 20% of patients in the high stimulation arm had more seizures.1

Concerns about safety

The safety of the device hinged on the cause of death among 17 patients of the 1000 implanted with the device worldwide. The device had been approved in Europe and Australia before approval in the US. The most common cause of death among young epileptics is sudden unexpected death from epilepsy (SUDEP), a poorly understood complication that is thought to be related to cardiac or respiratory arrest that sometimes occurs shortly after a seizure. SUDEP is reported most often among younger patients and those with poorly controlled seizures.2 3 The company told FDA panellists that the deaths were from SUDEP or other causes unrelated to its product.1 However, one panellist, Steven Piantadosi, professor of oncology and biostatistics and a clinical trial methodologist at Johns Hopkins, expressed concerns, saying, “I’m still a little worried about the death rates that we are seeing.”

In the data provided by Cyberonics, patients implanted with the device had a SUDEP rate of 7.3/1000. Dr Piantadosi asked, “Should we be concerned by that?”

In response to Dr Piantadosi’s questions, Ann Costello, an FDA reviewer, cited one epidemiological study that showed an even higher SUDEP rate, 9.3/1000, among treatment resistant patients about to have brain surgery to treat their severe seizures.4 W Allen Hauser, a member of Cyberonics’ scientific advisory board, also responded to Dr Piantadosi’s concerns, saying, “I don’t think that the sudden death is an issue specific to the device. It’s a specific issue in terms of people with bad epilepsy.”

Conditions of approval

Dr Piantadosi continued to express concern about the SUDEP rate, but the advisory panel nonetheless voted unanimously to approve the vagus nerve stimulator for patients with treatment resistant partial seizures. The FDA made the approval conditional: Cyberonics would have to conduct a post-approval study to examine the safety of the device and it would have to report promptly all serious adverse reactions to the agency.5

In the 13 years since the device was approved in the US, more than 900 deaths have been reported to the FDA, and it is still not clear what impact, if any, the device has had on patient mortality. Although Cyberonics conducted post-approval studies, none of the studies submitted to the FDA included mortality data. The FDA did not specifically require Cyberonics to submit mortality data as part of the follow up study, merely to “characterize morbidity and mortality.” According to a spokesman for Cyberonics, the company did collect mortality data from the Social Security Death Index, but they have not published the results and they declined to show the data to the BMJ. The FDA told the BMJ that it has not requested further studies concerning mortality.

Problems with post-approval surveillance

The FDA’s failure to request and rigorously monitor mortality data related to the VNS is but one example of the gap in post-approval surveillance of medical devices. Most devices and drugs on the market are supported by studies that are underpowered to detect rare but potentially life threatening events that can kill tens of thousands of people if the drug or device is widely used.6 The impracticality of conducting large scale clinical trials before approval for every drug and device places a burden on post-approval surveillance.

This burden is especially important for devices because they are less likely than drugs to be supported by clinical studies before use. Less than one third of devices approved under FDA’s premarket approval process had been evaluated in a randomised study, according to a review of 78 high risk cardiovascular devices approved by the FDA from 2000 to 2007.7 Just 5% were supported by two or more blinded, randomised studies. Most of the outcomes measured, according to the FDA study, were surrogate markers.

To monitor safety of devices, the FDA relies on reports of harms associated with devices once they have been approved. According to the agency, the most “comprehensive source of information about the safety and effectiveness” of devices as they are used in everyday circumstances is its Manufacturer and User Facility Device Experience (MAUDE) database.8 However, the data contained in the database does not constitute a comprehensive record of the numbers of adverse events or exposures, and thus can only pick up “signals” of possible safety problems that could be used to trigger more definitive investigation (box). Of course, these signals can be picked up only if someone is monitoring the data. An FDA taskforce on device regulation concluded in August, however, that “challenges” in “current data sources . . . make it difficult to . . . effectively obtain complete information about the risks and benefits of regulated products.”9

The FDA’s ability to detect potentially unsafe devices is further hampered by the fact that many post-approval studies required as a condition of the device’s approval are not conducted or conducted so poorly as to be meaningless. In 2005, the FDA evaluated the quality of post-approval studies. Susan Gardner, director of the Office of Surveillance and Biometrics at the Centers for Device and Radiological Health, which oversees approval and safety of devices, told an FDA advisory panel that 45 of the 127 premarket approvals granted between 1998 and 2000 had orders requiring post-approval surveillance or studies. But more than one fifth of the studies, she said, couldn’t be evaluated for quality because there was no record at the FDA showing they had ever even been conducted.13 FDA has since automated its records but many companies continue to submit data deemed “inadequate” by the agency, while other studies remain unreported.

Warning signs

So what does the MAUDE database tell us about the vagus nerve stimulator? Most MAUDE death reports, which are mainly written by manufacturers, contain descriptions so brief that no definitive cause of death can be inferred. Despite the paucity of information available (narratives are often just a sentence or two in length), the device is often exonerated. For example, one death associated with the VNS was reported to be from SUDEP on the basis of a “visual autopsy.” Marcia Angell, a pathologist and former editor in chief of the New England Journal of Medicine, says that without an actual autopsy it is impossible to determine whether it was SUDEP or some other factor, including potentially the device itself, that caused the patient’s death, and even then causality could remain uncertain.

Some narrative descriptions, however, are clear enough that further investigation of the device would seem warranted. For example, on 4 December 2008, a patient with a vagus nerve stimulator was admitted to hospital after an abrupt increase in seizures and was found to have “severe asystolia” coincident with vagus nerve stimulation. The device was disabled and the asystole ceased. The database has more than a dozen reports in which the device’s activity seems to be associated with asystole.

The database also contains reports of what could conceivably be downstream effects of asystole and bradycardia, such as increased “seizures,” new onset drop attacks, and fainting—which in turn could be responsible for some instances in which patients were reported to have died in a fall, by drowning, or in a road crash.

Another case that ostensibly should have prompted further investigation by the FDA, was that of 48 year old Dennis Fegan. According to medical records obtained by the BMJ with Mr Fegan’s permission, he woke up in pain at 2 am on 2 July 2006, only to pass out. After regaining consciousness, Mr Fegan, who had a 14 year history of partial complex seizures, passed out several more times. He thought he was having an unusually bad run of seizures. When an ambulance crew arrived, they witnessed Fegan having a seizure and administered intravenous diazepam. Once he was at the hospital and connected to a cardiac monitor, an apparent cause of his seizures emerged; at 3 minute intervals, his heart stopped for 30 seconds—synchronous with the firing of his vagus nerve stimulator. As soon as the device was disabled, the asystole ceased. His neurologist, a consulting cardiologist, and the treating emergency physician concluded that the device was the likely cause of his asystole.

Denominators, study bias, and failure to report

The FDA cautions that the MAUDE database “is not intended to be used either to evaluate rates of adverse events or to compare adverse event occurrence rates across devices.”14 Its purpose is to allow FDA analysts and others to pick up “signals” of harm that can trigger further investigation. Any investigation would need to evaluate the numerator (number of adverse events) and the denominator (number of exposures), but this information is routinely absent.

In the case of the vagus nerve stimulator the denominator depends not only on the number of patients implanted with the device but also on how many devices are still functioning. Cyberonics reports that by 10 March 2010, they had received registrations for 57 284 patients worldwide implanted with the device, yet it acknowledges that it is impossible to know how many patients have had their devices deactivated because it has no way to collect these data. The combination of possible under-reporting of adverse events (the numerator) and possible over-reporting of the number of active devices (the denominator) could mean that the rate of deaths among device users is higher than is apparent. The more than 900 deaths among relatively young people (most of those with implants are 15 to 44 years old) 3 did not trigger a request for further investigation.

And what of the post-approval study, known as XE05, ordered by the FDA as a condition of approval? A spokeswoman for the FDA cited XE05 as part of the evidence that “supported the long term safety and effectiveness of the device.” Cyberonics confirmed that only 50 patients were enrolled in the open label study and deaths were not recorded. When asked how such a small study that didn’t include mortality data could show the device’s “long term safety,” Cyberonics replied that “the purpose of that study wasn’t to look at SUDEP or mortality rates” but to evaluate other long term safety issues.

The FDA gave the BMJ references to five additional post-approval studies as evidence of the device’s safety. Yet the five studies do not establish that the device wasn’t responsible for deaths because none of them included mortality data. The largest study consisted of 4743 patients in the company’s outcome registry. The other studies evaluated subsets of registry patients. Cyberonics acknowledged that “mortality was not an endpoint collected as part of the Epilepsy Patient Outcomes Registry.”15

In 2005, the FDA approved the vagus nerve stimulator for the treatment of depression, despite the unanimous recommendation against approval by its own scientists. The FDA experts were concerned, in part, about the device’s safety. Nicholas Stine, who was then a research associate at Public Citizen, a public interest group, wrote to the agency, urging that the device should not be approved for treating depression. He and his coauthor, Peter Lurie, now adviser to FDA’s assistant commissioner for policy, said that “numerous reports” of worsening depression, suicides, and sudden deaths during clinical trials of vagus nerve stimulation “had not been adequately investigated,” and that the manufacturer “had not demonstrated long term safety of the device.”16 However, Cyberonics told the BMJ: “The FDA concluded that Cyberonics provided ‘reasonable assurance of safety and effectiveness’ of VNS therapy for the treatment of depression based on valid scientific evidence. There is no evidence linking VNS therapy to worsening depression, suicides, or sudden deaths.”

The company has also suggested that the stimulator might have a role in treating obesity, stroke, traumatic brain injury, and other conditions, and has previously taken out patents for these potential therapies.17

Potential solutions

The gaps in post-approval monitoring of the vagus nerve stimulator are emblematic of the FDA’s surveillance of all devices. Many of the problems have relatively easy fixes. Although the MAUDE database is flawed, it can still serve its purpose if qualitative signals are detected and trigger better analysis and quantification of a potential safety issue. Manufacturers could be required to regularly submit denominator data, which could be facilitated by requiring companies to provide a website link that patients or their surgeons could use to report whether the device is disabled or removed. An independent review panel could be appointed to decide whether adverse outcomes could be excluded from reporting. And, as the FDA has suggested, mechanisms to limit widespread uptake of new devices could be put in place so that fewer patients are harmed if it ultimately turns out that newly approved devices are flawed.