Authors

Mr. Peter McCulloch
Professor of Surgical Science and Practice
University of Oxford
Email: peter.mcculloch@nds.ox.ac.uk

Evidence-based surgery: do we need it, and can we get it?

Since the inception of the evidence based medicine movement, there have been regular attacks on surgeons for the perceived deficiencies of the evidence base in their speciality, and their apparent complacency in the face of this¹. Surgeons have not always been their own best advocates in responding to this criticism, but some have attempted to mount a defence. This has comprised first an investigation of the true state of the surgical evidence base, and second a review and discussion of the particular features of surgery as a discipline which pose problems for randomised controlled trials. An analysis of general surgical work in a large UK hospital showed that only 24% of the treatments used were based on RCT evidence, compared with over 50% for inpatient general medicine². A recent analysis of the illnesses and treatments most commonly encountered in general surgery suggested that less than 40% of operative treatments were amenable to study using an RCT design³. Some of the reasons suggested for this, such as the rarity or emergency nature of the conditions involved, are not wholly convincing, but others are important. It is useful to separate the historical features of this debate from those which continue to affect the prospects for a satisfactory evidence base for surgery in the future.

Historically, it has to be accepted that the majority of common general surgical operations were introduced before 1920, long before the importance of the randomised controlled trial was appreciated anywhere in the medical profession. On the other hand, most of the modern therapeutic armamentarium of the physician, dates from the era since World War 2, when pharmacology underwent a parallel and symbiotic growth with RCT methodology. It is always hard to do randomised trials of well established treatments, because the attachment of both doctors and patients to the familiar prevents the level of open-minded doubt necessary to achieve “equipoise”, that condition of uncertainty which allows a doctor ethically to randomise her patient between competing treatments. For this reason many operations (together with time-honoured medical treatments like morphine have largely escaped the rigours of the RCT versus placebo for their original indications.

So much for the history. The nature of treatment by surgical operation does however provide at least two good reasons for not performing RCTs (as well as a number of bad ones). Many of the conditions treated by surgery are of a mechanical nature, and in some cases (such as relief of mechanical bowel obstruction) the superiority of the mechanical solution offered by operation over non-treatment is self-evident. In many surgical scenarios, the benefits are so clear that no-one would consider a trial ethical or remotely sensible. Thus there was, and is, no question of a placebo-controlled trial of repair of inguinal hernia (in patients fit enough to be considered for one), relief of mechanical bowel obstruction or drainage of abscesses.

Surgery is a skilled, multistep process, and this makes RCT designs difficult to deliver in surgical studies, for two reasons. First, there is a learning process in every new operation, even for a fully trained surgeon unfamiliar with the particular procedure. Serious bias can easily be introduced if this is not acknowledged and measured or eliminated, especially for trials of new versus older procedures. Second, there is inherent variation in the way in which the procedure is performed by each different individual, and this cannot be eliminated. Surgeons stress the need for quality control in the technical aspects of any procedure under trial, but are acutely aware of the difficulty of the task. As in other areas in which the practitioner can be regarded as part of the treatment, it may be appropriate to stratify surgical trials by surgeon: this alone is not enough, however. Good surgical quality control means having valid objective measures which can demonstrate that an operation has been carried out according to predefined principles. This is a great deal more demanding than measuring patient compliance with drug treatment, but even more important: until recently, it was largely absent. Much of the cynicism expressed by surgeons about RCTs stems from their concern about this inability of crude designs to acknowledge the critical importance of quality in defining surgical outcome. We know that large variations in outcome are observed between surgeons performing similar operations in the same population⁴. If these do indeed reflect differences in the quality of surgery, it is salutary to note that they may be significantly greater than the expected differences between groups from ancillary treatments such as chemotherapy.

So surgery can be said to have a less secure evidence base than internal medicine, albeit for understandable reasons. If we look forward rather than back, this poses two questions. First, how do we evaluate and use an evidence base which is currently poor in RCTs, but rich in treatments for which a placebo-controlled trial would be unthinkable? Second, how can we create a more secure base for the future, taking into account the real problems outlined above? Whether we need to tackle these questions is no longer a question worth asking. The public demand for evidence-based justification of medical practice in all specialities is becoming more and more starkly expressed in the actions and statements of the courts and of political leaders. It is no longer possible to appeal to the mystique of professional expertise, when asked to justify our decisions – or our results. The less secure the evidence base for our practice, the less likely it is to be able to withstand pressure from public and political voices. The changes demanded by those with a better understanding of public opinion than of medical practice are unlikely to be beneficial in the long run. Surgeons, like other doctors, need evidence-based medicine now, because the alternative is policy-based medicine.

So we need evidence-based surgery. Can we get it? Yes, but it will be significantly more difficult than in some other branches of medicine. To address the first of the questions raised above, we need to take a serious and systematic look at the best way to evaluate what we have in terms of surgical evidence. We wish we had more RCTs, but we don’t. It will take years to reverse this situation, and until then we must either make the best efforts we can to analyse the available evidence from largely non-randomised studies or adopt an attitude of staunch agnosticism. I believe the former is the more sensible approach, but it represents a significant development in the scope of evidence based medicine. To achieve it we need rules, themselves well researched and validated, for determining the quality of the non-randomised studies in which surgery abounds. We need to develop a system of description which allows us to make an estimate of the efficacy of a treatment based on the best of this data, together with an estimate of the relative quality of this evidence. This is the best we can hope to achieve from the starting point we have been given, and it is better than nothing. We surgeons will need help in delivering it, particularly from clinical epidemiologists and statisticians. In order to provide this, many of them will need to put aside misgivings about supporting what they will probably see as a validation of second-class evidence. Using what we have should not be seen as such, and the best guarantee to those with concerns should be the efforts of surgeons to improve the quality of clinical surgical research in the future.

There is an important task to be undertaken in defining the current state of the evidence in surgery, and in particular the areas in which there is both an important unanswered question and a realistic means to performing the studies to answer it. As we define the most important areas of ignorance, we need to develop the tools to investigate them. There are ways to address the practical difficulties peculiar to surgery, and they need to be taken seriously and instituted. It may require log-books for consultant surgeons, plus site visits from co-ordinating experts, but it is, in principle, possible to overcome the problems of the learning curve. Satisfactory quality control may need similar unfamiliar measures, such as detailed guidance to define what variations in technique are acceptable within the trial, and submission of case notes, videos or x-rays to confirm correct technique. All of this will make surgical trials much more difficult and expensive to run. If this is to succeed it will need political support for the funding of a clinical research infrastructure with the resources to do the job properly. This is perhaps the most difficult obstacle of all, but it must be faced. It would make no sense to take the alternative approach of failing to resource the development of better surgical trials, and then criticising surgeons for not performing them. Give us the tools and we will finish the job.

Peter McCulloch
Liverpool, April 1999-04-13

Sample scenarios, searches, completed worksheets and CATs for evidence-based general surgery

Choose one of the following scenarios:

• diagnosis
• prognosis
• therapy
• harm

References