AMA Journal of Ethics^®

Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations.  BMJ. 2004;328(7454):1490-1494.

Proposal of new methodologies for writing clinical practice guidelines is now a common occurrence. Many groups and organizations, including the American Medical Association and the Institute of Medicine, have proposed careful procedures for creating guidelines.^1,2,3,4 The underlying objective of practice guidelines is to assist clinicians and patients in making difficult health care decisions. Sometimes even seemingly simple decisions require the assistance of guidelines (see, for example, the 3 clinical cases in this issue of Virtual Mentor). The need for guidelines arose partly from concerns over unexpected variation in medical care,^5,6 and from the desire to control health care costs.⁷

As more groups have begun using practice guidelines, critiques of their methodologies have also evolved. Some studies have expressed concern that guidelines do not adhere well to their methodological standards.^8,9 Another concern is that guideline recommendations are not timely due to infrequent revision.¹⁰ Finally, the focus of early guidelines on the effectiveness of interventions may have confused users that the strength of a recommendation is contingent upon the magnitude of a clinical response rather than upon the strength of supporting evidence that a clinical response does or does not exist.¹¹

Concerned about this current health care environment, not only in the United States but other countries as well, the GRADE working group has proposed a new process for writing practice guidelines.¹² According to their Web site,¹³ the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Working Group is an international collaboration of people intent on creating a common, sensible approach to practice guidelines. Their previous work has examined the use of numbers and symbols to communicate grades of evidence from existing guidelines.¹⁴ Now the GRADE group has moved beyond the communication of existing information to propose a comprehensive and systematic methodology for creating new practice guidelines.

After opening with a convincing explanation of the need for practice guidelines, the GRADE article deftly outlines the difficult choices that practitioners and patients must routinely make when faced with clinical decisions—choosing which potential outcomes to consider, what evidence is important to each outcome, and how to judge the quality of that evidence. The authors criticize the current environment where the presence of several competing guideline systems with differing methodologies causes confusion. The article then illustrates the many positive features of practice guidelines, such as the potential to prevent errors and disseminate clinical information.

To accomplish these goals, the authors propose an open system for making judgments about the quality of evidence and the strength of recommendations. For example, the GRADE system requires reviewers to rigorously examine 4 key and sequential elements when judging the quality of evidence: study design, study quality, consistency, and directness. Doing this, they maintain, should lead to a concise and explicit statement about the quality of evidence.

The next major consideration in the GRADE system is the strength of recommendations. The article highlights the trade-offs between benefits and harms inherent in making a recommendation, acknowledging that the outcomes and their seriousness may vary greatly depending on the patients' clinical histories and social environment. Partly to address these issues, the GRADE clinical guidelines would be specific to patient groups and practice settings. While the GRADE system stresses that the foremost considerations are health benefits and harms, it also suggests the role of incremental health care costs. Unfortunately, like many other guideline systems, the GRADE proposal offers little in response to the vexing questions of when or how such costs should be considered. One of its aims, however, is to make the process more transparent and open to public scrutiny, and this is a laudable goal.

The GRADE system is a thoughtful approach to an acknowledged problem. Writing clinical guidelines challenges the authors to make simple and concise recommendations while still retaining enough complexity to be useful under a variety of individual circumstances. The GRADE system proposes a specific and lengthy process to help retain the needed complexity while ultimately producing a clear recommendation. The process clearly identifies all decision points, allowing others to more fully evaluate each guideline.

The authors make clear that their article is only a summary of their methodology; some important questions remain unanswered. For example, the article does not discuss the selection of panel members to write the guidelines. In a system with explicit methodologies for making recommendations, determining panel composition also requires clear procedures. The identity of panel members greatly influences how users view the guidelines. Furthermore, the article does not address the problem of time. An ideal guideline system would adjust recommendations to changes in technology and the practice environment over time, but no such process is identified in this article. Arguably, the process by which guidelines are updated is as significant as that by which they're developed; out of date guidelines may, in some cases, be worse than no guidelines at all. A final concern is deciding which outcomes are most important to consider. Although the article provides admirable methodologies for evaluating given outcomes, choosing which outcomes are relevant is a value judgment that would benefit from more discussion.

These caveats are not flaws as much as areas in need of amplification. The real issues are not procedural limitations but rather the perceived need for a new system. The article proposes a strong process for writing clinical guidelines, but it is not obvious that this system is vastly superior to any of the systems already in use. The presence of multiple guideline systems is confusing. Having one standardized system for writing guidelines would be beneficial, and the GRADE system is admirable. Of more use than designing a new system, however, are articles that specifically compare and contrast existing systems with the ultimate goal of standardizing one approach over all others. After choosing one process for writing clinical guidelines, future discussions can shift from technical issues of methodology to the more clinically relevant difficulties inherent in medical decisions.