For a reappraisal of mechanistic evidence by the Health Technology Assessment (HTA) bodies

For a reappraisal of mechanistic evidence by the Health Technology Assessment (HTA) bodies

by Pr Jacques MASSOL

At the bottom of the EBM hierarchy, mechanistic evidence is not highly valued by the HTA bodies responsible for evaluating therapies, either for drawing up recommendations for clinical practice (for individual decisions) or for collective decision-making (market access for the therapies in question, various public health decisions).

This is partly due to the disappointments that followed the marketing of dangerous drugs based on mechanistic theories and trials carried out using intermediate criteria (the most emblematic example is that of anti-arrhythmics), and also to the lack of any guarantee that knowledge of the mechanisms involved is complete. Mechanistic studies still come up against certain difficulties. For example, mechanisms may change over time – how can these changes be identified (e.g. the appearance of resistance mutations in oncology) – or they may be masked. Furthermore, as the links between the elements of the causal chains are rarely deterministic, the probabilities linking them are often not available. The evaluator is therefore cautious about the ability of mechanistic reasoning to predict a therapeutic effect, and even more about its ability to predict the quantity of this effect on humans.

Yet the EBM hierarchy of evidence does not do justice to the importance that mechanistic evidence should have in the acquisition of evidence in therapeutics. There are various reasons for this.

The first is logical. Indeed, to say that a therapy T produces an effect E in a patient or a group of patients, it is necessary to be able to demonstrate a causal link between T and E. To do this, we have two types of scientific evidence: those from studies that seek to demonstrate a probabilistic relationship of dependence between T and E, conditional on possible confounding factors directly in the individual patient; these are therapeutic trials (if possible RCTs) placed at the top of the EBM evidence scale. On the other hand, very diverse studies carried out in vitro, in vivo or in silico, aimed at elucidating the mechanisms[1] by which T produces E: these are the so-called mechanistic studies. Neither type of evidence is, as a rule, logically sufficient to affirm causality. Indeed, what distinguishes causal correlations from non-causal correlations is the existence of the mechanism by which the occurrence of T explains the occurrence of E. By blindly valuing clinical causal correlations over mechanistic evidence, even though both are necessary[2], EBM can therefore cut out part of the demonstration if it discards mechanistic evidence[3].

 Thus, when the issue is not to sort the literature into levels of evidence for the strength of individual clinical studies (in which case a hierarchy can be defended), but to determine how strong the evidence is, the evidence from RCTs and mechanistic evidence should be placed side by side rather than compared. This is what many philosophers of medicine have been advocating for years and is now expressed in the so-called EBM + movement, which proposes tools for evaluating mechanisms in medicine (http://ebmplus.org/wp-content/uploads/2018/07/EBM-tools-v01.pdf).

Among the many authors seeking to revalorize the place of mechanisms in the foundation of medical evidence, we should mention Borgerson who argued in 2008 for methodological pluralism (Borgerson, 2008), Russo & Williamson, 2007 for their famous RWT complementarity thesis, Illari, 2011, La Caze, 2011, Clarke, et al, 2013, ….). They do not all agree on this place and on the questions that could be answered by mechanistic reasoning (especially for performing extrapolation of RWT results to a target population) (see J Howick’s position), or for individualizing the decision from RWT data (see H Andersen’s position, 2013), with R Bluhm, 2013, agreeing with neither, but they all consider mechanistic evidence and reasoning to be underutilized within the dominant EBM paradigm (we include here the methods used by HTA organizations).

However, it would be excessive not to recognize some virtues of EBM. And it will be acknowledged that its monolithic vision of EBM evidence has generated simple and useful tools for evaluators to answer a certain type of question. However, it must be admitted that in terms of mechanistic reasoning, it has also caused a conceptual regression. In particular, the EBM paradigm’s treatment of mechanistic evidence has offended the causal arguments described by Sir Bradford Hill in 1965, which are still dear to epidemiologists (Bradford Hill 1965). Indeed, when the question is to determine the extent to which evidence of causality is constituted, it is not so much a matter of deciding on the contribution of this or that type of study but of gathering as much different evidence as possible in favour of causality. Nine types of indicators of causality, which can be related to one of the two types of evidence mentioned above, were described by Bradford-Hill and constitute the foundations of analytical epidemiology: (1) strength of association; (2) consistency of the observed association; (3) specificity of the association; (4) temporality (the cause occurs before the effect); (5) biological gradient (the dose-response curve); (6) biological plausibility; (7) coherence ‘with the generally known facts of the natural history and biology of the disease’; (8) experimental evidence; (9) analogy. Six of these arguments are considered to be mechanistic (items 4,5,6,7,8,9 are strong indicators of an underlying mechanism), five probabilistic (items 1,2,3,5,8 are strong indicators of correlation) (Russo and Williamson, 2007). While Hill does not tell us how to assemble these indicators to obtain the best epistemic guarantees of the causal relationship at stake, it is accepted that the more arguments of each type that are accumulated in favour of a causal link, the more certain the causal relationship is. Thus, depending on the issue, HTA agencies might benefit from abandoning vertical hierarchical reasoning and replacing it with horizontal methodological pluralism in order to assemble the set of probabilistic and mechanistic arguments needed to construct the evidence for causality.

A second, much more practical reason is that the mechanisms of HTA do not contextualize the application of their hierarchy to the circumstances, or rather, when they do, do not explain how. In medical practice, it is not a matter of saying what is true with certainty but “what works” or “what might work. The question is not what is the best evidence of causality? Rather, we are asking the practical question: is the evidence of causality between T and E sufficient to convince us that therapy T will be useful for patient X or deserves to be reimbursed by the community?  In this respect, the dogmatic overbearing position of applying hierarchical intangible rules is not tenable in all circumstances. The relative importance of mechanistic evidence and RCTs depends on the question being asked and the possibilities for acquiring the best evidence. If insulin were discovered today, knowing the mechanistic surrogate properties of this hormone, would there be a need for a placebo-controlled trial in patients with acetowhite coma?

Ideally, of course, most of the time, we would have both convincing results from several coherent, high-powered, unbiased trials plus valid and complete mechanistic theories, but there are many cases where the evidence from RCTs is uncertain, insufficient, or even where no comparative trials are available, and yet the decision must be made, whether individually or collectively. These situations are diverse: trials that are difficult to perform (rare diseases or increasingly frequent splitting of frequent diseases: how to perform an RCT in patients with a rare EGFR mutation?) or good reasons to accelerate the availability of innovative health products (fast track, early access programs, compassionate uses) ….

In all situations where there is a strong, sometimes urgent, unmet therapeutic need, HTA agencies are faced with a difficult dilemma: authorize a therapy on the basis of uncertain evidence, at the risk of misjudgment, or refuse it while waiting for more solid evidence, at the risk of a possible loss of chance for patients. While it is not our place to comment on the role that mechanistic evidence could play in helping to manage the uncertainty generated by incomplete clinical evidence, we will allow ourselves to claim that such a situation only merits discussion if there is a strong mechanistic theory supporting it.

The divergence of opinion among international HTA agencies, with the same (weak) evidence for early COVID treatments, has recently triggered public unease with these bodies, not so much because their opinions diverged but because, more or less clinging to their hierarchy of evidence, they did not make explicit the elements of their deliberation, including the place of mechanistic arguments in their decision.

A third and more current reason for revaluing mechanistic evidence and revising the EBM hierarchy of evidence is that this hierarchy and the very notion of mechanistic evidence standing side by side with clinical trial evidence is outdated. A common perception was that the two types of evidence could not be combined either in bedside clinical practice or in EBM-type assessments by HTA agencies (see Nardini et al, 2012)[4].

Established before the digital era with the conviction that mechanistic and probabilistic evidence could only be separated, the hierarchy of evidence has remained fixed, without taking into account new methods of knowledge acquisition and more systemic conceptual models of the pathophysiological and pharmacological mechanisms involved. The current wealth of mechanistic models and the deep integration of these models with in silico RCT should allow both individual and collective predictions. Subject to their empirical validation, these models could thus contribute to redesigning the methods of assessment of HTA and to revise the idea that it would be impossible and inappropriate to combine evidence from trials with mechanistic elements. Finally, such models could provide valuable assistance to clinical practice by helping to answer one of the main questions in medicine: how to use general scientific knowledge in the case of a particular individual?

It is therefore time to revalue mechanistic evidence and to dust off the EBM hierarchy of evidence.

---