The Lancet : COVID-19 vaccine efficacy and effectiveness—the elephant (not) in the room

01/06/2021

Outcome Reporting Bias in COVID-19 mRNA Vaccine Clinical Trials 

Effectiveness of vaccines is less straightforward than it might seem. Depending on how the effect size is expressed, a quite different picture might emerge. Read the publication at The Lancet Microbe.

Relative Risk Reduction (RRR) vs Absolute Risk Reduction (ARR)

Abstract   

Vaccine efficacy is generally reported as a relative risk reduction (RRR). It uses the relative risk (RR)- ie, the ratio of attack rates with and without a vaccine-which is expressed as 1-RR. Ranking by reported efficacy gives relative risk reductions of 

  • 95% for the Pfizer-BioNTech
  • 94% for the Moderna-NIH
  •  90% for the Gamaleya, 67% for the J&J, and 
  • 67% for the AstraZeneca-Oxford vaccines. 

However, RRR should be seen against the background risk of being infected and becoming ill with COVID-19, which varies between populations and over time. Although the RRR considers only participants who could benefit from the vaccine, the absolute risk reduction (ARR), which is the difference between attack rates with and without a vaccine, considers the whole population. ARRs tend to be ignored because they give a much less impressive effect size than RRRs: 

  • 1·3% for the AstraZeneca-Oxford 
  • 1·2% for the Moderna-NIH 
  • 1·2% for the J&J, 0·93% for the Gamaleya and 
  • 0·84% for the Pfizer-BioNTech vaccines

There are many lessons to learn from the way studies are conducted and results are presented. With the use of only RRRs, and omitting ARRs, reporting bias is introduced, which affects the interpretation of vaccine efficacy. When communicating about vaccine efficacy, especially for public health decisions such as choosing the type of vaccines to purchase and deploy, having a full picture of what the data actually show is important, and ensuring comparisons are based on the combined evidence that puts vaccine trial results in context and not just looking at one summary measure, is also important. Such decisions should be properly informed by detailed understanding of study results, requiring access to full datasets and independent scrutiny and analyses.

Unfortunately, comparing vaccines on the basis of currently available trial (interim) data is made even more difficult by disparate study protocols, including primary endpoints (such as what is considered a COVID-19 case, and when is this assessed), types of placebo, study populations, background risks of COVID-19 during the study, duration of exposure, and different definitions of populations for analyses both within and between studies, as well as definitions of endpoints and statistical methods for efficacy. 

Importantly, we are left with the unanswered question as to whether a vaccine with a given efficacy in the study population will have the same efficacy in another population with different levels of background risk of COVID-19.  This is not a trivial question because transmission intensity varies between countries, affected by factors such as public health interventions and virus variants. The only reported indication of vaccine effectiveness is the Israeli mass vaccination campaign using the Pfizer-BioNTech product. Although the design and methodology are radically different from the randomised trial,2 Dagan and colleagues11 report an RRR of 94%, which is essentially the same as the RRR of the phase 3 trial (95%) but with an ARR of 0·46%, which translates into an NNV of 217 (when the ARR was 0·84% and the NNV was 119 in the phase 3 trial). This means in a real-life setting, 1·8 times more subjects might need to be vaccinated to prevent one more case of COVID-19 than predicted in the corresponding clinical trial. 

Uncoordinated phase 3 trials do not satisfy public health requirements; platform trials designed to address public health relevant questions with a common protocol will allow decisions to be made, informed by common criteria and uniform assessment. These considerations on efficacy and effectiveness are based on studies measuring prevention of mild to moderate COVID-19 infection; they were not designed to conclude on prevention of hospitalisation, severe disease, or death, or on prevention of infection and transmission potential. Assessing the suitability of vaccines must consider all indicators, and involve safety, deployability, availability, and costs.