A Critical Review of Outcomes
Gregorio A. Sicard, MD, explains the origin of the SVS Outcomes Committee and its current initiatives, including examination of high-risk patient data from the US IDE endovascular aneurysm repair trials.
Endovascular Today: What prompted the formation of the SVS steering committee regarding outcomes in endovascular repair for high-risk aneurysm patients?
Dr. Sicard: Originally, there was the Lifeline Registry Steering Committee spearheaded by the Society of Vascular Surgery. The concept was that we should have registries or data acquisitions that could provide outcomes analyses. The EuroStar registry, which was a different type of registry, has been very successful in endovascular aneurysm repair data analysis. We thought that the US should also have a similar registry, especially since there were IDE trials being done. Industry agreed to provide their IDE trial data, which is the same data they submitted to the FDA and consequentially has been highly audited.
Eventually, the Lifeline Registry was able to publish the results of the five IDE trials in 2005. Afterwards, it was expanded to include second-generation and newer-generation devices that were becoming available. As we reorganized the Lifeline Registry, we believed that a better way of accumulating this data was to create the SVS Outcomes Committee, because the issues surrounding outcomes had moved further than just having registries. In the future, we will get paid based on outcomes; CMS is very interested in the concept of pay for performance (P4P).
We decided to form the SVS Outcomes Committee, which would still receive the data submitted from industry. In addition, we would like to add other outcomes data, such as carotids, lower extremities, renals, and other vascular conditions. It was decided that the newly formed SVS Outcomes Committee would be the group that would spearhead the acquisition and the analysis of outcomes in vascular diseases.
Endovascular Today: Recently, the SVS Outcomes Committee examined the endovascular aneurysm repair data from the US IDE registries for high-risk patients. Why was this done?
Dr. Sicard: In 2005, Roger Greenhalgh, MD, presented the EVAR-2 trial results, which shook up our group because the perioperative mortality rate of 9% seemed exceedingly high. We were under the impression that US results were nowhere near a 9% perioperative mortality rate and believed that it would be beneficial to evaluate the high-risk patients from the IDE trials and determine how they compare with those of EVAR-2. The EVAR-2 article does not describe the inclusion criteria, but refers to another article published in the European Journal of Vascular and Endovascular Surgery that provided the inclusion criteria for both EVAR-1 and EVAR-2.
We then extracted all of the criteria used for the EVAR-2 trial, and went back to the approximately 3,000 patients who were in the Lifeline registry of the IDE trials and looked at the number of patients who met the same criteria. We were surprised to find that 565 patients met the high-risk criteria, which was far more patients than in EVAR-2. Although the IDE registries were not part of randomized trials, we believed that this was a strong dataset because it had the very extensive auditing that goes on with FDA evaluations of any device. Also, we recognized that the IDE trials were not comparing endovascular repair to surveillance, as did the EVAR-2 trial. The SVS Outcomes Committee decided to compare endovascular aneurysm repair to open surgical controls that met the same high-risk criteria. Not surprisingly, we found that there were fewer open cases that met the same high-risk criteria.
Endovascular Today: When you examined the EVAR-2 and US IDE data, what did you find?
Dr. Sicard: We found that there was a significant difference in 30-day mortality–2.9% in the IDE registries–when compared to 9% in EVAR-2. In addition, when looking closely at the published EVAR-2 data, a few things stood out. A significant number of patients in the treatment arm died prior to treatment. Nine of the 20 deaths occurred in the treatment arm due to aneurysm-related death before the patients were treated, but after randomization. The mean waiting period for the treatment arm was 57 days. In addition, of the 172 patients in the surveillance (or no-intervention) arm of the trial, 47 crossed over to the endovascular aneurysm repair arm.
Finally, the 4-year survival rate for our IDE group was 56% (based on all-cause mortality, not just aneurysm-related mortality), whereas in EVAR-2, the 4-year survival rate was 36%, so there was difference of 20 percentage points between the two trials.
Endovascular Today: Based on the objective criteria available, how closely did the patients in EVAR-2 and the US IDE registries match?
Dr. Sicard: We used the available criteria in terms of age, size of the aneurysm, cardiac disease, including coronary artery disease, congestive heart failure, chronic obstructive pulmonary disease and renal malfunction, renal abnormalities, and renal dysfunction. We tried to match the patients on as many data points as were available in the two studies. There is a table in our recent article that compared the patients in the EVAR-2 with the US IDE patients, in terms of cardiac, pulmonary, and renal function. Based on those criteria, the two studies matched pretty well.
Endovascular Today: Why was there such an extensive waiting period in the endovascular treatment arm of the EVAR-2 study?
Dr. Sicard: It is somewhat unknown what the basis was for this waiting period; however, it has been explained that some of these patients were in such bad medical condition that they needed immediate medical attention to get the patient in better shape prior to endovascular treatment. We have not seen any publication of those 20 patients showing the details regarding the nine patients who died during this waiting period.
As we stated in our analysis, the way EVAR-2 was set up and the way the data were analyzed (using a small number of patients with a significant number of deaths prior to treatment in the endovascular repair arm, and a significant numbers of patients crossing over from the surveillance arm) created a negative result for endovascular repair.
Endovascular Today: Did your committee conduct an analysis of the EVAR-2 data, attempting to compensate for these deficiencies?
Dr. Sicard: Well, if you don't have the raw data it's hard to do that. Yes, you could calculate that if they had 20 deaths and you eliminate the nine deaths that occurred prior to treatment, then their mortality would be almost cut in half, and a similar adjustment could be made for the surveillance arm of patients who crossed over to the treatment arm. However, that does not explain the significantly higher all-cause mortality in the EVAR-2 trial.
Endovascular Today: Did you have other issues with the design of the EVAR-2 trial?
Dr. Sicard: Yes. We have some issues with the way they set up the trial. This trial was not limited to an objective list of high-risk criteria, but also employed a "pragmatism rule," in which the internists, cardiologists, radiologists, or surgeons were required to make a determination if the patient was unfit for surgery. That is not a very objective manner of analyzing or setting up a trial because it lacks uniformity in what is truly a high-risk patient. You cannot measure pragmatism.
Endovascular Today: What is the explanation for the large difference in both aneurysm-related and all-cause mortality between EVAR-2 and the US IDE data?
Dr. Sicard: We can speculate on the explanation for this difference, but there is no good way of proving it unless you analyze every patient who lived and every patient who died (that applies to their medical condition and risks). One explanation that the EVAR-2 trial investigators have raised is that they were dealing with sicker patients. Another possible explanation is that the actual medical care of these patients was different. Perhaps the patients in the EVAR-2 trial did not have the same access to their doctors as those in the US. I don't know if this is the case, but it's a possibility that should be explored. The only way to give credibility to any particular explanation for the differences in results between the IDE trial and the EVAR-2 trial is to analyze the raw data for both groups.
Endovascular Today: What prompted the quick publication of the US IDE data?
Dr. Sicard: Right around the time that we were analyzing the EVAR-2 trial publication, the SVS was informed that the Agency for Healthcare Research and Quality (AHRQ) was putting together a document to evaluate endovascular aneurysm repair. However, we knew the impact that this type of analysis had in Europe because of the results of a Belgian document that evaluated endovascular aneurysm repair. In that document, the extensive analysis of endovascular repair, which included quality of life, economics, etc., revealed that endovascular aneurysm repair had no long-term benefit. That document also stated that similarly there was no benefit of endovascular repair in high-risk patients based on one publication that addressed this question (the EVAR-2 trial). At this point, there was nothing in the literature challenging this conclusion. The SVS Outcomes Committee decided to look at the Lifeline Registry IDE trials data and determine how many high-risk patients were in this database. This was the impetus for our analysis.
AHRQ is a government agency that is part of the US Department of Health & Human Services and works very closely with CMS. AHRQ was already performing an extensive analysis of endovascular repair. The SVS was provided with a first draft of the AHRQ document for comments. Similar to the Belgium report, the AHRQ draft questioned whether endovascular repair is indicated at all for high-risk patients. The SVS Outcomes Committee responded that the AHRQ document should not be published prior to evaluating the results of our high-risk patient analysis. The SVS paper had just been submitted the Journal of Vascular Surgery, but it had not yet been published. AHRQ does not consider any review of data that has not been published. They were kind enough to withhold their publication for 1 month, which gave us the necessary time for the review process. Within this period of time, the article was accepted and published on the Web. AHRQ acknowledged receipt of the Web publication and that it would be considered in their review of endovascular aneurysm repair.
Endovascular Today: Could this AHRQ report affect CMS's decision to reimburse the procedure?
Dr. Sicard: I believe so. That possible affect is looming. Especially as AHRQ and CMS start working together to develop methods for linking outcomes to payment.
Gregorio A. Sicard, MD, is Professor of Surgery, Chief of General and Vascular Surgery, Washington University School of Medicine, St. Louis, Missouri. Dr. Sicard may be reached at (314) 362-7841; firstname.lastname@example.org.