Influenza vaccination: policy versus evidence

Policies

Every vaccination campaign has stated aims against which its effects must be measured. The US Advisory Committee on Immunisation Practices produces a regularly updated rationale for vaccination against influenza.1 The current version identifies 11 categories of patients at high risk of complications from influenza (box).

The rationale rests on the heavy burden that influenza imposes on the population and the benefits of vaccination. For example, reductions in cases, admissions to hospital, mortality of elderly people in families with children, contacts with healthcare professionals, antibiotic prescriptions, and absenteeism for children and household contacts are the main arguments for extending vaccination to healthy children aged 6-23 months in the United States.2 Canada introduced a similar policy in 2004.3 Less comprehensive policies recommending vaccination for all people aged 60 or 65 and over are in place in 40 of 51 developed or rapidly developing countries.4 On the basis of single studies, the World Health Organization estimates that “vaccination of the elderly reduces the risk of serious complications or of death by 70-85%.”5 Given the global nature of these recommendations, what type of evidence should we expect to support them and what does available evidence tell us?4

Which evidence?

When considering the best evidence for vaccination we must take into account the unique epidemiological features of influenza viruses and the rationale for immunisation. The incidence and circulation of seasonal influenza and other respiratory viruses vary greatly each year, each season, and even in each setting. A systematic review of the incidence of influenza in people up to 19 years' old reported a seasonal variability of 0-46%; during a five year period the average incidence was 4.6% in this age group. During a period of 25 years the incidence was 9.5% in children under 5.6 Because of this variability and lack of carryover protection from one year's vaccine to the next,7 especially if the virus changes its antigenic configuration, single studies reporting data from one or two seasons are difficult to interpret. Single studies are also not reliable sources for generalising and forecasting the effects of vaccines, especially when numbers are small. They introduce further instability into already problematic forecasting. Additional limitations to our forecasting ability are imposed by our use (and misuse) of studies assessing the effects of influenza vaccines. Although the effect assessed depends on the aims of the particular campaign, most concentrate on serious effects (such as pneumonia or death) and person to person transmission (table 1). Field efficacy studies are only relevant when viral circulation is high, but no one can forecast with precision the impact on next year's influenza.

Flu vaccination: is the effort justified?

Credit: SATURN STILLS/SPL

• Download figure
• Open in new tab
• Download powerpoint

Studies of the effects on influenza-like illness and its complications most closely replicate real life conditions because no one knows what agent (if any) causes this disease. Influenza-like illness is an acute respiratory disease caused by many different viruses (including influenza A and B), which presents with symptoms and signs that cannot be distinguished from those of influenza. Influenza-like illness does not have documented laboratory isolation of the causative agent and is the syndrome that most commonly presents to doctors (“the flu”).

In general the most powerful and reliable studies are those that “average” out several years and perform subanalyses by setting, population, viral circulation, and viral-vaccine antigenic match—variables that affect interpretation of the effects of a vaccine. Systematic reviews are the best way to perform such analyses, and provide powerful evidence weighted by the methodological quality of the studies involved. Large datasets containing several decades of observations help us to assess the performance of vaccines more accurately.

The evidence

I searched for relevant systematic reviews when updating and expanding the Clinical Evidence chapter on influenza (see bmj.com)—evidence was plentiful. The examples in table 2 show the strength of the evidence and the contradictions in relation to the stated aims of the vaccination campaign. Whenever possible, I chose evidence gathered in the optimal circumstances (for inactivated vaccines)—high viral circulation and a good match between the viral antigen and the vaccine.

Three problems are immediately apparent. The first is heavy reliance on non-randomised studies (chiefly cohort studies), especially in the elderly. This makes assessment of methodological quality an important part of data interpretation. For example, of 40 datasets assessing the effects of influenza vaccines in elderly people in institutions, only 26 reported data on viral types in circulation and only 21 gave information on vaccine content. Insufficient data were available in 11 of 17 retrospective studies of elderly people in institutions to allow reviewers to assess the authors' claim of “high” or “epidemic” viral circulation.¹¹¹⁴ A metaanalysis of inactivated vaccines in elderly people showed a gradient from no effect against influenza or influenza-like illness to a large effect (up to 60%) in preventing all-cause mortality. These findings are both counterintuitive and implausible, as other causes of death are far more prevalent in elderly people even in the winter months.¹⁵¹⁶ It is impossible for a vaccine that does not prevent influenza to prevent its complications, including admission to hospital.

A more likely explanation for such a finding is selection bias, where one half of the study population (hemi-cohort) systematically differs from the other in one or more key characteristics.^14–16 In this case, the vaccinated hemi-cohort may have been more mobile, healthy, and wealthy than the control hemi-cohort, thus explaining the differences in all-cause mortality.¹¹¹⁴ The same effect is seen in stronger study designs (such as cluster randomised trials) that are badly executed, which introduces bias.10 Its presence seems to be a marker of confounders that persist even after adjusting for known ones, and it makes accurate interpretation of the data difficult. Caution in interpretation should thus be the rule, not the exception. This problem (in the opposite direction—with frailer people more likely to be vaccinated) has been identified before but not heeded.17 The only way that all known and unknown confounders can be adequately controlled for is by randomisation.

The influence of poor study quality is also seen in the outcome of a review of evidence supporting the vaccination of all children to minimise transmission to family contacts.18 Five randomised studies and five non-randomised studies were reviewed, but although data were suggestive of protection, its extent was impossible to measure because of the weak methods used in the primary studies.18

The second problem is either the absence of evidence or the absence of convincing evidence on most of the effects at the centre of campaign objectives (table 2). In children under 2 years inactivated vaccines had the same field efficacy as placebo,8 and in healthy people under 65 vaccination did not affect hospital stay, time off work, or death from influenza and its complications.9 Reviews found no evidence of an effect in patients with asthma or cystic fibrosis, but inactivated vaccines reduced the incidence of exacerbations after three to four weeks by 39% in those with chronic obstructive pulmonary disease.¹²¹³¹⁹ All reviewers reported small data sets (such as 180 people with chronic obstructive pulmonary disease13), which may explain the lack of demonstrable effect.

The third problem is the small and heterogeneous dataset on the safety of inactivated vaccines, which is surprising given their longstanding and widespread use. A Cochrane Database Systematic Review found only one old trial with data from 35 participants aged 12-28 months.8 In the general population of elderly people, despite a dataset of several million observations, safety was only reported in five randomised controlled trials (2963 observations in total) on local and systemic adverse events seen within a week of giving parenteral inactivated vaccine.11 Although there appears to be no evidence that annual revaccination is harmful, such a lack of knowledge is surprising.

Gap between policy and evidence

The large gap between policy and what the data tell us (when rigorously assembled and evaluated) is surprising. The reasons for this situation are not clear and may be complex. The starting point is the potential confusion between influenza and influenza-like illness, when any case of illness resembling influenza is seen as real influenza, especially during peak periods of activity. Some surveillance systems report cases of influenza-like illness as influenza without further explanation. This confusion leads to a gross overestimation of the impact of influenza, unrealistic expectations of the performance of vaccines, and spurious certainty of our ability to predict viral circulation and impact. The consequences are seen in the impractical advice given by public bodies on thresholds of the incidence of influenza-like illness at which influenza specific interventions (antivirals) should be used.20

The confusion between influenza and influenza-like illness is compounded by the lack of accurate and fast surveillance systems that can tell what viruses are circulating in a setting or community within a short time frame, and after the “season” is finished give an accurate picture of what went on to enable better forecasting of future trends.21 Accurate surveillance must be based on a properly worked out sampling system for cases of influenza-like illness that meet set criteria, with accurate and quick feedback of a presumptive microbiological diagnosis. Without this, we cannot generalise from random sampling.

Another reason may be “availability creep.” In their efforts to deal with, or be seen to deal with, policy makers favour intervention with what is available—registered influenza vaccines. A similar philosophy is the “we have to make decisions and cannot wait to have perfect data” approach. This attitude may have an altruistic basis but has two important consequences. Firstly, it uses up resources that could be invested in a proper evaluation of influenza vaccines or on other health interventions of proven effectiveness. Secondly, the inception of a vaccination campaign seems to preclude the assessment of a vaccine through placebo controlled randomised trials on ethical grounds. Far from being unethical, however, such trials are desperately needed and we should invest in them without delay. A further consequence is reliance on non-randomised studies once the campaign is under way. It is debatable whether these can contribute to our understanding of the effectiveness of vaccines. Ultimately non-randomised designs cannot answer questions on the effects of influenza vaccines.

The optimistic and confident tone of some predictions of viral circulation and of the impact of inactivated vaccines, which are at odds with the evidence, is striking. The reasons are probably complex and may involve “a messy blend of truth conflicts and conflicts of interest making it difficult to separate factual disputes from value disputes”22 or a manifestation of optimism bias (an unwarranted belief in the efficacy of interventions).23

Whatever the reasons, it is a sobering thought that Archie Cochrane's 1972 statement that we should use what has been tested and found to reach its objectives is as revolutionary now as it was then.