Prospective clinical performance studies – Worth the effort?

In the IVDR-era, clin­i­cal per­for­mance stud­ies have gained much impor­tance. In fact, this expen­sive but nec­es­sary step in IVD-devel­op­ment is one of the most broad­ly dis­cussed obsta­cles for man­u­fac­tur­ers to overcome.
One of the most fre­quent­ly asked ques­tions: Should a clin­i­cal per­for­mance study be ret­ro­spec­tive or prospective?
First, let’s take a look at what these terms actu­al­ly mean. Sup­pose that we need 100 Flu A pos­i­tives and 200 Flu A neg­a­tives for the val­i­da­tion of an influen­za A rapid test. In a ret­ro­spec­tive study, we would specif­i­cal­ly pro­cure sam­ples from patients who are known Flu A pos­i­tive or Flu A neg­a­tive (e.g., by pur­chas­ing pre-analysed sam­ples from a biobank or by fresh­ly col­lect­ing sam­ples from donors with known clin­i­cal condition).

[Info box: Care­ful: in study design, ret­ro­spec­tive does not nec­es­sar­i­ly mean that left­over sam­ples are used. Ret­ro­spec­tive stud­ies can be con­duct­ed with fresh sam­ples as well, if the clin­i­cal con­di­tion of the donor was known.]

In a prospec­tive study, the main dif­fer­ence is that we do not know the clin­i­cal con­di­tion of the donor before they are includ­ed in the study. Donors would be select­ed accord­ing to oth­er inclu­sion cri­te­ria, e.g., cer­tain symp­toms. If we were to include 300 donors with flu-like symp­toms, we would not be able to pre­dict how many of them would be pos­i­tive or neg­a­tive. The out­come might only be 44 pos­i­tives and 256 neg­a­tives because a lot of donors were infect­ed with oth­er res­pi­ra­to­ry pathogens. In such a case, the col­lec­tion must be con­tin­ued until the desired num­ber of Flu A pos­i­tive sam­ples is acquired. There­fore, the total nec­es­sary sam­ple num­ber for prospec­tive stud­ies is usu­al­ly high­er than for ret­ro­spec­tive studies.
So why both­er with prospec­tive study designs? From a pure­ly aca­d­e­m­i­cal point of view, prospec­tive stud­ies gen­er­ate much more valid data. The sep­a­ra­tion of pos­i­tives and neg­a­tives in a ret­ro­spec­tive study design is prone to selec­tion bias due to the inclu­sion of clear­ly pos­i­tive or neg­a­tive sam­ples. Con­se­quent­ly, we are mak­ing it easy for our index test (= the test that is to be val­i­dat­ed). In a prospec­tive study design, we include sam­ples from donors who may be just bare­ly pos­i­tive or who are pos­i­tive for sim­i­lar bio­mark­ers that could lead to false-pos­i­tive results. Here, we are prop­er­ly chal­leng­ing our index test.
Be that as it may, the effort, time, and bud­get that is nec­es­sary to have a prospec­tive clin­i­cal per­for­mance study con­duct­ed, is sub­stan­tial. There­fore, many man­u­fac­tur­ers accept the down­sides of a ret­ro­spec­tive study design to the ben­e­fit of quick­er and cheap­er results.
Care­ful, though: It is expect­ed that in the near future this ‘easy way’ will not be viable anymore.
Accord­ing to IVDR arti­cle 57.2, ‘where appro­pri­ate, per­for­mance stud­ies shall be per­formed in cir­cum­stances sim­i­lar to the nor­mal con­di­tions of use of the device’. For most IVDs, the nor­mal con­di­tion of use would be in con­se­quence of a cer­tain med­ical indi­ca­tion. This implies a study design in which the true con­di­tion of the donors is unknown dur­ing test­ing with the index test, hence a prospec­tive design. One of the main goals of the IVDR is that man­u­fac­tur­ers pay heed to this logic.
While the expres­sion ‘where appro­pri­ate’ seems to give us some lee­way, it is becom­ing appar­ent that the leg­is­la­tors’ pref­er­ence for prospec­tive stud­ies will be more and more enforced by sec­ondary legal acts.
A first exam­ple is EU com­mon list of COVID-19 anti­gen tests, pub­lished by the Euro­pean Com­mis­sion Direc­torate-Gen­er­al for Health and Food Safe­ty. This doc­u­ment con­tained require­ments for SARS-CoV­‑2 assay val­i­da­tions which were de fac­to bind­ing for the indus­try. Notably, it also dif­fer­en­ti­at­ed between two cat­e­gories of assays: Cat­e­go­ry A list­ed only prod­ucts that had been val­i­dat­ed through prospec­tive clin­i­cal field stud­ies, while Cat­e­go­ry B con­tained those with ret­ro­spec­tive in vit­ro stud­ies. EU Mem­ber States were ‘strong­ly encour­aged’ by the authors to use only Cat­e­go­ry A devices for COVID test cer­tifi­cates which exclud­ed Cat­e­go­ry B devices from a major reim­burse­ment opportunity.
We expect that oth­er future pub­li­ca­tions will go in the same direc­tion. While it may not become ille­gal to con­duct ret­ro­spec­tive per­for­mance stud­ies, the pref­er­ence for prospec­tive stud­ies by key opin­ion lead­ers, expert pan­els or gov­ern­men­tal author­i­ties will often­times make this more com­plex and expen­sive study design the only option for manufacturers.
Long sto­ry short: It is tempt­ing to facil­i­tate clin­i­cal val­i­da­tion through a ret­ro­spec­tive study design. How­ev­er, the risks that are asso­ci­at­ed with this deci­sion should not be neglect­ed. The time will come when ret­ro­spec­tive clin­i­cal data is sim­ply not enough any­more. A prospec­tive study may cost more and take longer but you gain first-rate data and are safe from any future enforce­ment actions.