Audit and Feedback Interventions

Robbie Foy, University of Leeds
Martin Eccles, Newcastle University

Download: Presentation (PDF, 202KB)

Overview

Chart audit
- Rationale and what it is
- Types of audit criteria
- Target setting
- Practical issues
Audit and feedback
- Rationale and what it is
- Theory
- Evidence
Summary and future research directions

Key learning points

Measuring adherence to clinical practice recommendations can highlight important implementation gaps and inform subsequent priorities for knowledge transfer
Audit and feedback can be effective in improving professional practice although the effects on clinical practice are generally small to moderate
More research is needed on the effects of audit and feedback compared to other interventions, and the mechanisms by and contexts in which it works best

Chart audit

Rationale
- Recognised gaps and delays in the implementation of evidence based practice
- Can be identified or confirmed by chart audits
What it is
- Documented clinical care is measured against a review criterion: “a systematically developed statement that can be used to assess the appropriateness of specific healthcare decisions, services and outcomes.”

Field MJ, Lohr KN. Guidelines for clinical practice. From development to use. Institute of Medicine. Washington: National Academy Press, 1992

Types of review criteria

Explicit criteria
- Aim to maximise the reliability and objectivity of measurement
- E.g. patients newly diagnosed with essential hypertension who either have persistent high blood pressure (BP) of 160/100 mmHg or more, or are at raised cardiovascular risk with persistent BP of more than 140/90 mmHg have a record of being offered drug therapy
Implicit criteria
- Involve peer or expert clinicians making judgements about the desired care
- Tend to be more subjective and less reliable than explicit criteria
- Mainly used to assess complex processes of care or adverse outcomes, e.g. maternal deaths related to childbirth

Naylor CD, Guyatt GH, for the Evidence-based Medicine Working Group. How to use an Article About a Clinical Utilization Review. JAMA 1996;27518:1435-9

Structure of healthcare delivery
- E.g. the presence of calibrated devices for measuring blood pressure
Healthcare processes
- E.g. the prescription of anti-hypertensive medication
- Structural and process criteria must be valid, i.e. strong evidence exists that their improvement is associated with improvement in outcomes of care
Patient outcomes
- Short term or surrogate outcomes, e.g. blood pressure levels
- Long term outcomes, e.g. stroke
- Tend to be less sensitive at detecting changes in practice because many factors may influence patient outcomes
- Generally require more resources, larger sample sizes, and longer follow up to detect important changes

Target setting

Target levels of performance can be set to guide subsequent decisions on whether implementation activities are worthwhile
Attempts to improve already high levels of performance may not be as productive as switching attention to alternative priorities
Why not to expect 100% adherence
- For many clinical actions, there is a “ceiling” beyond which healthcare systems’ and clinicians’ abilities to improve performance are limited
- E.g. if eligible patients prefer to avoid drug treatment or experience unacceptable side effects

Practical issues

Planning and conducting reviews
- Sampling procedures
- Sample size
- How to collect the data
Under-documentation
- Not all clinical actions are documented…
- But good documentation has become integral to good medical practice

Audit and feedback

Rationale
- Demonstrating the gap between actual and desired performance will motivate clinicians or healthcare systems to take action addressing that gap
What it is
- “Any summary of clinical performance of health care over a specified period of time” given in a written, electronic or verbal format

Jamtvedt G, Young JM, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2006(2)

Theory

“Self-regulation”
- A process of determining goals and then using these as reference values to bring existing states into line with those goals
- The success of any desired change also depends upon individuals being able to change their behaviour (e.g. clinical practice skills) or external influences on behaviour (e.g. organisational factors)

Carver CS, Scheier M. Attention and self-regulation : a control-theory approach to human behavior. New York: Springer-Verlag, 1981

Evidence

Cochrane Review of 118 randomised trials
- “Audit and feedback can be effective in improving professional practice and that the effects on clinical practice are generally small to moderate.”
- E.g. when percentage adherence with desired practice was measured, the effects ranged from a 16% decrease to a 70% increase, with a median improvement of 5%

Jamtvedt G, Young JM, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2006(2)

Why this variation in effect size?

Type of format, i.e. verbal, paper or electronic;
Frequency and duration, e.g. as a one-off step or continuously and often over a period of time
Source, e.g. whether from a supervisor or professional body
Content, e.g. information on healthcare processes or patient outcomes, use of identifiers to permit comparisons between individual professionals, teams or facilities
Use of various sources to deliver feedback, such as supervisors or professional bodies.

Uncertainties

Higher intensity of feedback
- Associated with greater effects but…
- Higher costs of intensive strategies may outweigh benefits
Combining audit and feedback with other strategies
- E.g. educational meetings
- Uncertainty as to whether increases effectiveness
Lack of head to head comparisons
- Of different methods of providing feedback and of comparisons of audit and feedback versus other interventions
- Therefore difficult to recommend the use of one intervention strategy over another on empirical grounds

The importance of context

Contextual factors
- Relative effects of audit and feedback greater when baseline adherence with recommended practice is low.
Nature of targeted behaviours
- Clinicians’ motivation to change practice and their level of engagement with the feedback intervention (whether they are active or passive recipients but…
- Effects of audit and feedback are greater for recommendations perceived by clinicians to be less compatible with current norms and for tasks associated with lower motivation

Foy R, MacLennan G, Grimshaw J, et al. Attributes of clinical recommendations that influence change in practice following audit and feedback. J Clin Epidemiol 2002;55:717-722

Palmer RH, Louis TA, Hsu LN et al. A randomized controlled trial of quality assurance in sixteen ambulatory care practices. Med Care 1985;23:751-70

Summary

Only limited insights into how and when audit and feedback can be made to work more effectively
Its selection as a KT intervention remains a matter of judgement based upon
- Current evidence base
- The working ‘diagnosis’ of the causes of an implementation gap
- Availability of supporting resources and skills
In principle, getting the diagnosis right offers a rational basis for choosing an approach to delivering feedback
- E.g. if perceived peer pressure was identified as a key determinant of clinicians’ practice or motivation to change for a given context, feedback could reasonably incorporate peer comparison

Research questions

By what mechanism or mechanisms does audit and feedback exerts its effects?
Which contextual features (e.g. setting, characteristics of healthcare professionals) and attributes of targeted clinical behaviours negate or enhance the effects of audit and feedback?
How does audit and feedback, by itself or in combination with other intervention, compare against other interventions to change clinical behaviour?

Foy R, Eccles M, Jamtvedt G, et al. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res 2005;5:50