In the IVDR-era, clinical performance studies have gained much importance. In fact, this expensive but necessary step in IVD-development is one of the most broadly discussed obstacles for manufacturers to overcome.
One of the most frequently asked questions: Should a clinical performance study be retrospective or prospective?
First, let’s take a look at what these terms actually mean. Suppose that we need 100 Flu A positives and 200 Flu A negatives for the validation of an influenza A rapid test. In a retrospective study, we would specifically procure samples from patients who are known Flu A positive or Flu A negative (e.g., by purchasing pre-analysed samples from a biobank or by freshly collecting samples from donors with known clinical condition).
[Info box: Careful: in study design, retrospective does not necessarily mean that leftover samples are used. Retrospective studies can be conducted with fresh samples as well, if the clinical condition of the donor was known.]
In a prospective study, the main difference is that we do not know the clinical condition of the donor before they are included in the study. Donors would be selected according to other inclusion criteria, e.g., certain symptoms. If we were to include 300 donors with flu-like symptoms, we would not be able to predict how many of them would be positive or negative. The outcome might only be 44 positives and 256 negatives because a lot of donors were infected with other respiratory pathogens. In such a case, the collection must be continued until the desired number of Flu A positive samples is acquired. Therefore, the total necessary sample number for prospective studies is usually higher than for retrospective studies.
So why bother with prospective study designs? From a purely academical point of view, prospective studies generate much more valid data. The separation of positives and negatives in a retrospective study design is prone to selection bias due to the inclusion of clearly positive or negative samples. Consequently, we are making it easy for our index test (= the test that is to be validated). In a prospective study design, we include samples from donors who may be just barely positive or who are positive for similar biomarkers that could lead to false-positive results. Here, we are properly challenging our index test.
Be that as it may, the effort, time, and budget that is necessary to have a prospective clinical performance study conducted, is substantial. Therefore, many manufacturers accept the downsides of a retrospective study design to the benefit of quicker and cheaper results.
Careful, though: It is expected that in the near future this ‘easy way’ will not be viable anymore.
According to IVDR article 57.2, ‘where appropriate, performance studies shall be performed in circumstances similar to the normal conditions of use of the device’. For most IVDs, the normal condition of use would be in consequence of a certain medical indication. This implies a study design in which the true condition of the donors is unknown during testing with the index test, hence a prospective design. One of the main goals of the IVDR is that manufacturers pay heed to this logic.
While the expression ‘where appropriate’ seems to give us some leeway, it is becoming apparent that the legislators’ preference for prospective studies will be more and more enforced by secondary legal acts.
A first example is EU common list of COVID-19 antigen tests, published by the European Commission Directorate-General for Health and Food Safety. This document contained requirements for SARS-CoV‑2 assay validations which were de facto binding for the industry. Notably, it also differentiated between two categories of assays: Category A listed only products that had been validated through prospective clinical field studies, while Category B contained those with retrospective in vitro studies. EU Member States were ‘strongly encouraged’ by the authors to use only Category A devices for COVID test certificates which excluded Category B devices from a major reimbursement opportunity.
We expect that other future publications will go in the same direction. While it may not become illegal to conduct retrospective performance studies, the preference for prospective studies by key opinion leaders, expert panels or governmental authorities will oftentimes make this more complex and expensive study design the only option for manufacturers.
Long story short: It is tempting to facilitate clinical validation through a retrospective study design. However, the risks that are associated with this decision should not be neglected. The time will come when retrospective clinical data is simply not enough anymore. A prospective study may cost more and take longer but you gain first-rate data and are safe from any future enforcement actions.
