Standardized Objective Performance Criteria and Control Datasets
How the FDA is working to determine the potential role for these criteria in peripheral device regulation, and how they are best established.
The views and opinions presented in this article are those of the authors and do not necessarily reflect those of the US FDA, the US Department of Health and Human Services, or the Public Health Service.
In previous issues of Endovascular Today, we have described some ongoing initiatives in the Peripheral Vascular Devices Branch intended to optimize our regulation of peripheral vascular devices. Several of these initiatives involve the establishment of standardized objective performance criteria (OPC) and control datasets for various peripheral devices, including iliac stents and endovascular grafts.
Level one evidence from a controlled clinical trial may be optimal for establishing the standard of care for a given patient population; however, this level of evidence is not always necessary to demonstrate that a particular medical device offers a reasonable treatment option for patients. Alternative study designs include single-arm comparisons to OPCs or control datasets. When using OPCs, a new device must be demonstrated to be no worse than the specified numbers in the OPCs, with a prespecified margin of error. In contrast, the standardized control dataset is used in place of prospectively collected control data, generally with the expectation that there would be an improvement in the new/investigational endovascular treatment as compared to the control dataset.
WHY ARE WE CONSIDERING ALTERNATIVE STUDY PLANS?
Conducting controlled clinical studies can be particularly challenging when the study device involves endovascular repair and the standard of care is either open surgery or using other endovascular devices off-label. These challenges include identifying both patients and clinicians who are willing to participate in the studies and defining appropriate controls. Given the potential for controlled studies to end up with incomplete enrollment and less-than-optimal control populations (eg, poorly matched to study patients, treated with off-label devices, or devices not representative of the current standard of care), alternative study plans are being considered.
WHY MAY THE ESTABLISHMENT OF STANDARDIZED OPCS AND CONTROL DATASETS BE REASONABLE OPTIONS?
Although iliac stenting is standard practice in the US, there are only four iliac stents approved for marketing. Although device manufacturers may prefer to market stents for an approved iliac indication, obtaining approval requires completing a clinical study, which is often plagued with slow enrollment due to off-label use of biliary stents. In addition, the rate of adverse events in these studies is generally very low, driving power calculations to impractical levels. These issues prevent device manufactures from completing studies in a timely and cost-effective manner, hindering technology from entering the market before it is archaic. Use of an OPC for iliac stents may promote more rapid completion of trials and provide valuable data to physicians and patients on current iliac stent technology. In addition, establishment of an OPC that could be used by all manufacturers would provide a consistent evaluation of these devices.
For endovascular grafts, however, it is not yet reasonable to consider establishing an OPC because the design, and hence the risk/benefit profile, for each of these devices is unique. Establishment of a standardized control dataset is being considered, based on what has been learned with the evaluation of these devices to date.
There have been five AAA endovascular grafts approved in the US, with safety and effectiveness demonstrated through a comparison to prospectively enrolled open surgical controls, with no less than 1 year of follow-up. Effectiveness has generally been based on the assumption that surgical repair is 100% effective and that the effectiveness of the endovascular graft will be no worse than a prespecified value. Safety has been compared using a composite safety endpoint and other secondary evaluations, including a comparison of parameters such as blood loss and lengths of ICU and hospital stays. As such, the primary purpose of the control dataset has been to evaluate the safety of the endovascular grafts through a comparison of serious adverse event rates in the control dataset.
From previous evaluations of endovascular grafts, it is evident that the primary problems with these devices include device integrity issues, such as fractures and graft wear holes, migration, sealing failures, and deployment problems. Assessment of the significance of these problems has not included a comparison to surgical control data because they are unique to endovascular repair. They may not be associated with serious adverse events, yet must be considered in the context of the potential to cause future events.
The establishment of a standardized control dataset for these devices may be reasonable because such a dataset should be adequate for comparing serious adverse event rates. In addition, there is a significant amount of open surgical repair data available for defining this dataset. The primary concerns with endovascular grafts might be better addressed with the enrollment of additional study device patients instead of the control patients.
HOW SHOULD STANDARDIZED OPCS AND CONTROL DATASETS BE ESTABLISHED?
For the OPCs and control datasets to be standardized, they would best be developed with collaboration between clinicians, manufacturers, and the FDA. Preferably, they would be proposed by the clinical community, with input from manufacturers and would be based on compiled results from previous studies or a meta-analysis of literature.
Ideally, for an iliac stent OPC, the study population should be clearly defined by establishing appropriate inclusion and exclusion criteria and the indications for treatment. The sample size should be driven by a hypothesis and should be statistically justified. Complication rates and success criterion should be supported by specific literature sources. This may be unattainable given the limitations inherent in the literature and data publicly available; however, a rigorous attempt should be made to establish a comprehensive and reliable dataset.
The following steps have been suggested for the development of the standardized control dataset for AAA endovascular grafts:
• Outline and define the important endpoints (eg, operative and 1-year mortality, serious adverse events, other adverse events)
• Adjudicate the individual datasets to apply standardized definitions for the compiled dataset
• Verify the proposed values (eg, with literature)
• Propose the control dataset to the FDA
FDA Advisory Panel review may be helpful in the recognition of an OPC or control dataset by the Agency. Once accepted, the OPC or dataset should be made available to all manufacturers.
WHAT IS THE STATUS OF THE DEVELOPMENT OF STANDARDIZED OPCS AND CONTROL DATASETS?
Both societies and individual physicians have developed proposals for OPCs for iliac stents; however, continued collaboration has not been established so that a consensus can be reached. Published intersociety consensus documents, meta-analyses, and reviews of the medical literature have been provided to the FDA to support an OPC, but there are still unanswered questions regarding the source of proposed values for complication rates and success criteria. In addition, the data provided are outdated and may need to be updated to reflect the current practice of medicine.
We are working with the Society of Vascular Surgery (SVS) to establish a control dataset for AAA endovascular grafts, based primarily on the SVS Outcomes Registry. Hopefully, this dataset will be proposed, possibly discussed by our cardiovascular advisory panel, and available to manufacturers within a year. For now, new studies for AAA endovascular grafts are proposing the use of such a control if available in time for the preparation of their premarket approval application, along with the collection of standard control data in case the dataset takes longer to establish than expected.
WHICH ISSUES WILL NOT BE ADDRESSED BY STANDARDIZED OPCS AND CONTROL DATASETS?
Although use of an OPC for iliac stent studies may make studies easier to complete, establishing an OPC will not eliminate the off-label use of biliary stents in the iliacs. In addition, given the rapid acceptance of iliac stenting as the standard of care, many clinical questions may always remain unanswered. However, the completion of well-controlled studies comparing iliac stenting to an OPC will provide additional evidence-based science to physicians that could ultimately improve the care of patients.
For endovascular grafts, establishment of a control dataset will allow for a comparison of serious adverse event rates between expected surgical repair results and a new endovascular graft; however, this study design will not address the question of aneurysm-related death. Frankly, a complete assessment of aneurysm-related death cannot be accomplished in the relatively short-term studies required for device approval in the US, regardless of the type of control used for these studies, although this endpoint is a focus of our postmarket surveillance. In absence of a thorough understanding of the risks of aneurysm-related death associated with endovascular repair of aneurysms, patients treated with endovascular grafts require close, continued, active, long-term follow-up to ensure timely interventions as needed to minimize the potential for aneurysm rupture.
Dorothy Abel and Angela Smith are Regulatory Review Scientists with the US FDA Center for Devices and Radiological Health in Rockville, Maryland. Ms. Abel is also a regular columnist for Endovascular Today. Ms. Abel may be reached at (301) 443-8262, ext. 165; firstname.lastname@example.org.