Advertisement
Advertisement
September 2011
What Did We Learn From the Learning Curve?
Both United States and European datasets suggest that practice makes perfect for carotid artery stenting.
“Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so.”
—Douglas Adams, Last Chance to See
In 2010 and 2011, we learned a great deal about the learning curve of carotid artery stenting (CAS). The CREST Abbott Vascular (Santa Clara, CA) premarket approval supplement presentation on January 26, 2011, at the US Food and Drug Administration (FDA) Circulatory System Devices Advisory Panel presented the Rx Acculink carotid stent system for consideration of an expanded indication, namely, for use in a standard operative risk population.1 In the presentation (including a total of 118 PowerPoint slides), a startling relationship between the adverse event rate and temporal inclusion in CREST was evident.
Within the last year to 18 months, there have also been United States2 (US) and European Union3 (EU) registry datasets that sought to evaluate the meaning of the learning curve for CAS. These were categorized as the influence of site and operator characteristics on CAS outcomes for the US dataset2 and an exploration of the relationship between experience and complication rates for the EU dataset.3 The data from these combined sources prove to be a very compelling argument in favor of the contention that “practice makes perfect” for the complex intervention that is represented by CAS.
RECAP OF ATTITUDES ON LEARNING CURVE FOR CAS ENCAPSULATED IN THE RECENT RANDOMIZED TRIALS OF CEA VERSUS CAS FOR STANDARD-RISK PATIENTS
EVA-3S
The investigators stated: “A potential bias in the comparison
of a relatively new procedure such as stenting with
an established procedure such as endarterectomy is the
effect of the learning curve. Our trial involved centers with
staff members who had various degrees of experience in
carotid stenting, including centers in which investigators
treated enrolled patients under the supervision of a tutor.
We tried to limit the effect of the learning curve through the careful training and supervision of interventional
physicians. We did not find any significant differences in
outcome related to the number of stenting procedures
performed in individual centers or to the experience of the
interventional physicians, although these analyses were
able to detect only large differences.”4
Regarding the French national trial, EVA-3S, comparing carotid endarterectomy (CEA) and CAS in standard-risk populations, the investigators concluded that experience did not influence outcomes for the CAS limb of the trial. The investigators consistently point out that there were no significant differences in outcomes between centers performing < 21 CAS procedures, those performing 21 to 40 CAS procedures, and those with > 40 CAS procedures under their belt. The trial was not powered to answer with confidence the question on learning curve for CAS.
Suffice to say that 85% of all operators performing CAS in EVA-3S had performed < 50 procedures, representing below entry-level requirements for any future randomized trial involving CAS. Meanwhile, the credentialing requirements for CEA within this trial were wholly different (ie, surgeons performing CEA had to have shown a minimum throughput of 25 CEAs in the preceding year, whereas CAS operators could have had a total experience of ≥ 12 CAS or ≥ 5 CAS and 30 non-CAS supra-aortic angioplasty or stenting procedures—an entirely different anatomic territory with different technical, lesion, and operator demands). Proctoring was allowed within the trial. The operator's first CAS case could be performed within the trial with a proctor that had performed as few as two cases.4,5
The background in terms of the reimbursement policy for CAS in France during the timeline of EVA-3S deserves special mention. A handful of French establishments were performing CAS at the start of the trial (perhaps as few as five). Because reimbursement could only be secured within the remit of a randomized trial, baseline expertise with CAS across France was severely limited, meaning that keen but motivated CAS operators would have to perform CAS within EVA-3S if they wanted to offer CAS at all, despite the fact that they were, in majority, wholly inexperienced. With the benefit of hindsight (ie, “the retrospectoscope”), we can see how naïve the statements made by the EVA-3S investigators really were.
SPACE
This German/Austrian/Swiss trial of CEA versus CAS for
standard-risk populations “failed to show the noninferiority
of CAS” (largely because it was prematurely halted on
the grounds of futility—it did not run to completion—with
the additional death blow dealt by lack of further funding),
yet it also failed to show either the superiority of CEA or any significant difference between the two treatment
limbs in the primary powered outcome event of “any ipsilateral
stroke or death.”6
The initial requirements for CAS operators were more exacting than for the EVA-3S trial, as they required a minimum of 25 CAS procedures at entry level early in the trial. However, these entry requirements subsequently became more lenient after 2002, whereby entry after 10 CAS case experiences was allowable to boost recruitment. By 2002, less experienced operators were provided with a “preliminary certificate,” allowing them to perform CAS in the trial. Once again, the standards for surgeons offering CEA within SPACE were more exacting and remained resolutely so: a minimum of 25 CEAs were required with outcomes that had to meet preset morbidity and mortality thresholds.
The downregulation of entry criteria is well understood by many a beleaguered researcher: the recruitment rate and number became the Holy Grail of this (and many progenitor and ongoing trials). The original entry criteria for CAS within SPACE were comparable with CREST, and the outcomes for the primary endpoint were not statistically different between CAS and CEA; however, subsequent data from this trial showed overt and startling differences in outcomes for CAS in favor of treatment in more experienced CAS units compared to centers that were awarded a preliminary certificate. The relationship between center throughput and outcomes was remarkably linear.7
ICSS
At the Transcatheter Cardiovascular Therapeutics
2010 annual meeting, during a personal communication
among myself, Dr. William A. Gray, and the Principal
Investigator of ICSS, Prof. Martin Brown (University
College, London), Professor Brown was adamant that to
support the generalizability of ICSS trial results, a relatively
lax attitude to CAS operators' experience was tolerated in
this United Kingdom–based international randomized
trial of CEA versus CAS in standard-risk patients. The
general contention was that the trialists did not want to
compare “expert stenters” with CEA as it is currently performed
within a wide generality of surgical units.8
Proctoring within the trial was allowed for CAS operators (I was one such proctor). The first CAS case could have been (and often was) performed within this trial with a proctor. After 10 cases, the operator was often signed off to fly solo. Meanwhile, the same lax attitude did not seem to apply to the surgeons within this trial, who were duty bound to have performed a minimum of 50 CEAs with an annual throughput of ≥ 10 per year, with the additional onus of meeting preset thresholds for stroke and death.8 Disparity in the experience levels of those performing CEA and those performing CAS within ICSS have engendered concerns about the ethics of valid comparisons within trial settings such as this.9
The ICSS investigators stated: “The risk of outcome events associated with stenting was lower in inexperienced, supervised centers than in more experienced centers … and there was no significant difference in the excess hazard of stenting compared with endarterectomy between supervised and experienced centers or between centers recruiting more or less than 50 patients; therefore, inexperience cannot explain our results.”
We must, however, bear in mind that the threshold determining experience amounted to 10 CAS cases—painfully low given our evolved understanding of the relationship between experience and outcomes for CAS. This was an audacious statement and one that was revised in light of further evidence from what we must remember was an interim analysis of ICSS and not the results of the primary endpoint analysis (ie, 3-year survival free of major disabling stroke, which is yet to be reported).
Subsequent analyses presented by the ICSS trialists (not yet published) revealed that in centers recruiting fewer than 50 patients, the stroke/death/myocardial infarction rate was 10%, and in centers recruiting > 50 cases, it was 5.9% (P = .035), meaning that there was no significant difference in outcomes between CAS and CEA in units recruiting more than 50 patients.10 This revelation does not come as any surprise for those units (like our own) that diligently randomized more than 100 patients into this trial.
CREST
CREST, through a careful lead-in phase and reflection on
procedural risks after CAS as documented in a number of
previous independently reviewed registries (with FDAmandated
stringent and independent adjudication of outcome
events) in high-risk populations, strove to ensure a
comparison of “like with like” with regard to outcomes for
CAS versus CEA within this trial. At Charing Cross Carotid
Question Time in April 2011, Dr. Michael Jaff outlined his
response to my query about the possible reduced generalizability
of CAS outcomes when performed by experts
within a trial as carefully proscribed as CREST.
Dr. Jaff explained, “I think it is right that in a trial like CREST, it's only responsible if you are trying an established therapy versus a ‘new' therapy to have those who are expert assess them, and once you prove that the technology has a role to play, then it's the responsibility of the medical community to figure out how to generalize the results out to the community. Actually, I'm fine with having highlevel experts with lots of experience in what's considered to be an important clinical trial.”11
Tantalizing results ensued from the CREST FDA Abbott Vascular panel. Despite the fact that CREST ensured highlevel operator expertise for CAS from its inception, there was still a clear pattern of differential outcomes, which reflected “within-trial learning” (Figure 1).1
This reduction in the major stroke/death rate for CAS patients within CREST from the first half to the second half of the trial in terms of the randomization time-line was, for the first time, elegantly displayed within a trial that is seen by many as exemplary in its conduct and process. It is an important stand-alone piece of evidence and is ultimately neoteric. Furthermore, scrutiny of the CREST dataset reveals similar findings for the outcome of “all stroke/death” and also specifically for the proportion of patients who were symptomatic at inclusion. Without these compelling data relating specifically to symptom status, it is easy to dismiss the finding of temporal improvement in outcomes as the result of increasing numbers of asymptomatic patients treated within the trial (their inclusion was allowed from the year 2005 onward), as it is well recognized that asymptomatic patients can expect a lower procedural hazard.
The FDA panel was very cognizant of learning curve issues—the FDA review (P040012/S3B) was careful to build an argument on learning curve issues and quoted EU datasets3 to support its argument. Such was their interest in learning curve issues that they were keen to consider additional postmarket surveillance studies that may further delineate operator experience for CAS.
REGISTRY DATA: US AND EU DATASETS AND AN EXPLORATION OF THEIR DIFFERENCES
The United States
Gray et al published an article entitled “Influence of site
and operator characteristics on carotid artery stent outcomes:
analysis of the CAPTURE 2 clinical study” in
February 2011.2 The dataset reported comprised 5,297
patients who were treated by CAS in 180 US hospitals by
459 operators between March 2006 and January 2009. The
final analysis was limited to 3,388 nonoctogenarian asymptomatic
patients. The embargo on the inclusion of symptomatic
patients in this analysis was as a result of the fact
that nonoctogenarian symptomatic patients comprised a
minority subset (721 of a total of 5,297 spread over
approximately 180 sites and 400 operators), meaning that
the outcomes of this subset would not be statistically
meaningful or representative of the much larger treated
population.
Furthermore, outcome events from the treatment of octogenarians were excluded—they again comprised a minority population (as they always have done in any randomized trial of CEA vs best medical therapy and in any trial of CEA vs CAS). Furthermore, Gray et al based their analysis on the American Heart Association (AHA) guidelines for 30-day death/stroke outcomes for CEA (these having not been primarily established for CAS) to define site and operator outliers. These AHA recommendations were based on predicate data that largely excluded the octogenarian population, as there were no accepted or established CEA thresholds for the octogenarian population (lacking adequate data), thus none could be used with confidence in the analysis presented.
The results of this sizeable, prospective, multicenter, independently adjudicated registry indicated that CAS could be performed in a pure dataset of asymptomatic nonoctogenarian patients with comfortably low allstroke/ death/myocardial infarction rates (2.9%) and allstroke/ death rates (2.7%). Notably, the potential confounders of stent type and type of embolic protection and their influence on outcomes were fortuitously controlled (the registry was limited to cases employing the Acculink stent and the Accunet filter [Abbott Vascular]).
The analysis showed a striking relationship between the regression lines representing the number of procedures per site and again for numbers of procedures per treating physician against outcomes. The adverse event rate fell linearly for the log-regression analysis for both parameters with increasing case number. The authors further stated that “a threshold of 72 cases was found to be necessary for consistently achieving a death/stroke rate below 3%,” which is consistent with the AHA guidelines for a comparable population. This threshold needs to be qualified given that it is derived from a single stent/embolic protection system, with operators who were mainly cardiologists, and in a relatively later era than many of the EU trials listed previously when there was already considerably more background experience with CAS. Other analyses that differ in these attributes might be expected to arrive at different thresholds.
The European Union
A systematic review published in 2010 sought to investigate
the evidence for the relationship between volume
and outcomes for CAS.3 Studies with > 100 interventions
that provided outcome data year by year were included.
The main outcome measure compared across studies was
all stroke/death. Where possible, comparable data were
pooled and analyzed using metaregression techniques. It
was not possible to perform a standard systematic review
and meta-analysis because of the lack of data from randomized
studies. When redundant studies were excluded,
four sizeable case series and one registry met the inclusion
criteria, numbering > 4,000 patient outcomes.
When the case series data were pooled, the χ2 test for trend showed a significant reduction in the combined stroke and death rate over time. Metaregression analysis of case series data allowed the setting of thresholds for acceptable stroke/death rates. Where year-by-year data were available, published stroke and death rates for CAS showed improvements over time. Although advances in technology and pharmacology may in part be responsible for improved latter-day results (also bearing in mind a steady increase in asymptomatic patients treated during the timeline of this review), temporal improvement in outcomes were shown in both early and contemporary cohorts. The consistency of the results strongly suggests the presence of a learning curve.
In active CAS units (with an average throughput of approximately 50 cases), it was clear that it may take almost 2 years before the stroke/death rates fell below an arbitrary 5% threshold. The absolute number requirements before an operator reaches the lofty heights of a 5% or 3% event rate seemed staggeringly high: an eye-watering median (95% confidence interval) of 196 (107–325) for a 5% event rate and 429 (304–609) for a 3% event rate.
Suffice to say that this has not been the case in my unit. The confounders are myriad and not easily corrected for. The included studies were weighted toward units with operators who had previously performed CEA, and arguably, these cardio(vascular) surgeons entered the CAS arena with perhaps a different skill set with regard to embolic protection and 0.014-inch guidewires, which are the mainstay of the CAS procedure, while other CAS operators (interventional radiologists, interventional cardiologists, and interventional neuroradiologists) could perhaps transfer their skill set to the carotid territory without too much wailing and gnashing of teeth.
A CRITIQUE ON THE DIFFERENCES BETWEEN THE US AND EU DATA
The US dataset reflects a pure population of asymptomatic nonoctogenarian patients treated with one stent and one embolic protection device. The outcomes were independently reviewed, and outcome events were independently adjudicated—a statistician's dream. The EU dataset was wholly more “dirty”—myriad CE Marked stents and variable use of CE Marked embolic protection devices within datasets that were sometimes not independently reviewed and with a mixed population (albeit with a growing asymptomatic populace to boot).
It should be noted that analyses such as this are influenced by the fact that they reflect early past decade versus later past decade comparisons—the pivotal trials fall victim to the same inherent problem to an extent (there were simply not enough qualified operators to run sizeable multicenter trials). CREST, by virtue of the time it took to reach its recruitment target, was slightly more resistant (although not immune) to the issue of operator experience on outcome.
Regardless, the take-home message is the same. There is a clear message from both datasets that outlines the importance of experience and throughput on outcomes.
Ultimately, we all appreciate that “you can't make an omelette without breaking a few eggs,” but why not embark on a good few cookery lessons in advance, such that our collateral “egg damage” is as limited as is humanly possible?
Sumaira Macdonald, MBChB (Comm.), FRCP, FRCR, PhD, is a Consultant Vascular Radiologist and Honorary Clinical Senior Lecturer at Freeman Hospital in Newcastle upon Tyne, United Kingdom. She has disclosed that she receives grant/research funding from Abbott Vascular, Bard Peripheral Vascular, Cordis Corporation, ev3 Inc., Gore & Associates, Medtronic Invatec, and Pyramed, and that she is a paid consultant to Abbott Vascular, ev3 Inc., Medtronic Invatec, and Gore & Associates. Dr. Macdonald may be reached at sumaira.macdonald@nuth.nhs.uk.
Advertisement
Advertisement
