December 27, 2004
Science Policy White House Bulletin Tells Agencies When They Should Peer-Review Scientific Data
The White House Office of Management and Budget issued final guidance December 17, 2004 on peer reviews of scientific information by federal agencies.
The Final Information Quality Bulletin for Peer Review encourages peer review of important scientific information generally and requires that two types of scientific information be peer-reviewed before the information is disseminated by the federal government.
The bulletin broadly defines scientific information to include "factual inputs, data, models, analyses, technical information or scientific assessments."
Scientific assessments include documents such as state-of-the-science reports; technology assessments; health, safety, or ecological risk assessments; toxicological characterizations of substances; integrated assessment models; hazard determinations; exposure assessments; or meta-analyses, which are a statistical method of combining the results of a number of different studies to produce a stronger conclusion than can be provided by any single study.
Two Types of Information
The two types of scientific information that must be peer reviewed are:
Although all influential scientific information must be peer reviewed, the bulletin gives agencies broad discretion in determining what type of peer review would be appropriate and what procedures should be employed to select appropriate reviewers.
The intensity of peer review should be commensurate with the import of the information that will be disseminated and its likely effect on policy decisions, OMB said.
The bulletin provides examples of different types of peer reviews that agencies could use, and it describes criteria agencies should use to select peer reviewers.
"The most important factor in selecting reviewers is expertise," OMB said. "Reviewers should also be selected to represent a diversity of scientific perspectives relevant to the subject."
However, agencies also must examine potential conflicts of interest, OMB said. "The National Academy of Sciences defines 'conflict of interest' as any financial or other interest that conflicts with the service of an individual on the review panel because it could impair the individual's objectivity or could create an unfair competitive advantage for a person or organization," OMB said, describing that definition as a useful benchmark for agencies to use.
Public participation in peer reviews can provide important information, OMB said. Participation can take a variety of forms including providing oral comments before a peer review panel or providing written comments to peer reviewers, the bulletin said.
The credibility of a peer reviewed document is likely to be enhanced if the public understands how an agency responded to concerns the experts raised OMB said.
"Accordingly, agencies should consider preparing a written response to the peer review report explaining: the agency's agreement or disagreement, the actions the agency has undertaken or will undertake in response to the report, and (if applicable) the reasons the agency believes those actions satisfy any key concerns or recommendations in the report," the bulletin said.
Some Assessments Warrant Rigorous Review
"Highly influential scientific assessments" require a more rigorous form of peer review than is required for influential scientific information, the bulletin said.
For example, peer reviewers of highly influential scientific assessments must prepare a report describing their conclusions, OMB said.
Federal agencies must prepare a written response to the peer review report, OMB said. Both the peer reviewers' conclusions and the agency's response, along with all materials related to the peer review, must be disseminated on the agency's Web site, OMB said.
The bulletin describes some "alternative procedures" federal agencies may use. For example, an agency may commission the National Academy of Sciences to peer review an agency product or employ some other procedure if that alternative is specifically approved by the administrator of OMB's Office of Information and Regulatory Affairs in consultation with the Office of Science and Technology Policy.
Federal agencies are required to create an "agenda," an Internet-accessible listing of documents that will be peer reviewed and procedures the agencies intend to use for that peer review.
Information to be provided for each item in the agenda includes:
Updates Every Six Months
Agencies should update their peer-review agendas at least every six months, OMB said, adding more frequent updates may be warranted for highly influential or other particularly important information.
Agencies are to develop and post their peer-review plans for highly influential scientific assessments within six months, OMB said.
Agencies have up to one year to develop and post their peer review plans for influential scientific information, OMB said.
By Dec. 15 of each year, agencies must give OMB's Office of Information and Regulatory Affairs a report summarizing the peer reviews it conducted during the previous fiscal year.
The report must include a variety of information including the number of times influential scientific information versus highly influential scientific assessments were reviewed and whether an agency asked to have its peer review requirements waived.
OMB's final guidance was revised based on two rounds of public comment and a workshop held Nov. 18, 2003.
The initial draft document, published in September 2003, was criticized by many different groups including Bruce Alberts, president of the National Academy of Sciences. Alberts was among many scientists who said the September 2003 draft was too prescriptive, might exclude scientists with appropriate expertise, and might prove counterproductive (2 DEN A-2, 01/6/04).
National Academy President Praises Guidance
In a statement issued Dec. 17, Alberts said: "The final document provides good guidance for facilitating high quality peer review of influential information disseminated by the government."
OMB had already addressed many of the academies' concerns when it issued a second draft document in April 2004, William Colglazier, executive officer of the National Academy of Sciences, told BNA.
The final version reflects "minor revisions' to that April iteration, OMB said.
"The final version looks fine; they've done quite a good job," Colglazier said.
Granger Morgan, who chairs the Environmental Protection Agency's primary peer review panel, the Science Advisory Board, agreed.
The history of OMB's peer review document illustrates the value of peer review, Morgan said.
"[OMB's guidance] has had a fairly long evolution, and the early version had some problems," Morgan said. "It was moderated in quite a few ways, and I think that has improved it," said Morgan, who heads of the Department of Engineering and Public Policy at Carnegie Mellon University.
Jim Solyst, who works on science policy issues at the American Chemistry Council, agreed. "This was clearly a well-vetted effort," said Solyst. All interested parties had ample opportunity to comment and concerns raised appear to be settled in the final document, he said.
Guidance May Affect Agencies Differently
The key issue now, said Colglazier and Morgan, is how well agencies implement the guidance.
Some agencies, such as EPA, already conduct the types of peer reviews required by OMB's bulletin and are unlikely to have to modify their procedures very much, Morgan said.
But peer review is no where near as well established in other agencies, Morgan said, adding that underscores the value of OMB setting standards and offering guidance.
Frederick Anderson, an attorney with McKenna Long & Aldridge LLP, said EPA's peer review practices can be improved.
But, compared to some agencies, EPA is doing a better job with peer review, Anderson said.
Agencies such as the Department of Energy and the Army Corps of Engineers "have everything to gain" from following OMB's guidance, Anderson said.
The final document achieves many of the goals of the original draft but in a far more streamlined fashion, Anderson said.
Benefits of Guidance Discussed
Some people may not realize how important certain aspects of OMB's bulletin are, Anderson said.
For example, when information is disseminated for peer review, OMB requires that the information include a statement noting that the document "does not and should not be construed to represent any agency determination of policy."
That statement is very valuable, Anderson said, adding it is not appropriate that regional EPA offices use draft documents for cleanup or other regulatory decisions.
OMB's definition of scientific assessment is so comprehensive, that Anderson said he could not think of a type of assessment that would not be covered.
Finally, OMB's requirements that agencies maintain public records describing what they will be peer reviewing and how provides interested parties many opportunities to weigh in on that process, Anderson said.
For example, if an agency does not intend to allow the public to present scientific concerns to a peer review panel, an interested party could ask the agency to do so, he said.
If an agency has classified a document as being influential scientific information and a party thinks the document is actually highly influential, than the agency could be asked to reclassify the information, Anderson said.
The Final Information Quality Bulletin for Peer Review will be posted on http://www.whitehouse.gov/omb/inforeg/infopoltech.html on the World Wide Web.