ECS Review of the Tennessee NBPTS Study Misses the Obvious
September 23, 2002
A Tennessee study of 16 teachers certified by the National Board for Professional Teaching Standards (NBPTS) found that none were producing exceptional student achievement gains. The Education Commission of the States (ECS) appointed a panel headed by Penn Education dean Susan Fuhrman to review it.
The just-released review both misses the central point of the study and prompts questions about the value of ECS policy recommendations.
The review focused primarily on whether the study’s subjects were an adequate sample of all NBPTS teachers.
In fact, the 16 subjects in the study were not treated as a sample. There was no attempt to generalize the findings to either the Tennessee or the national population of NBPTS-certified teachers. Rather the 16 subjects were the entire number of the NBPTS-certified teachers in Tennessee for whom value-added scores were available, i.e., teachers in grades 3-8. Precisely because the sample would have been too small, the study purposely avoided any conclusions about the national NBPTS population.
Unlike studies that sample a population and attempt to draw an inference about the larger group, the Tennessee study was simply a multiple replication trial of the NBPTS certification process. The value-added achievement gains of 16 NBPTS-certified teachers were examined. In 16 out of 16 cases they were found not to be exceptional producers of student achievement.
When a certification process is checked 16 times and found wrong every instance, any reasonable person would say it isn’t trustworthy regardless of what might be inferred about others who have been certified by the same process.
Here is an analogy: Suppose 100 football players are found to weigh 300 pounds when weighed on a bathroom scale, and 16 of them are taken to a doctor’s office and reweighed on a professional quality scale. If the 16 were found to weigh less than 250 on the doctor’s scale, the accuracy of the bathroom scale would be in doubt regardless of what might be presumed about the 84 who had not yet been reweighed.
Despite a four-month long review, the Fuhrman panel overlooked the obvious. Instead of being commissioned to examine the evidence on both sides of the NBPTS issue, it was given a narrow faultfinding assignment. As a result, it raised criticisms and questions that are beside the point. It is true that the Tennessee study does not answer a number of the panel’s questions, but the reason is that access to Tennessee’s teacher-effect data is tightly restricted and answers are mostly unavailable. In any case, Stone repeatedly offered to answer any question had by the panel but was ignored.
Other observers have commented favorably. For example, Stanford researcher Eric Hanushek has said, “Stone’s study follows a well conceived methodology.” The small number of teachers in his study, “ . . . is not Stone’s fault or choice. It simply represents the available universe of teachers.” “One thinks that, had Stone’s study of 16 teachers supported the certification program, it would have been widely publicized and little criticized.” (http://www.heartland.org/education/aug02/certified.htm)
Is ECS Helping Policymakers?
The nature and circumstances of the ECS review raises broader questions about the kind of advice that comes from ECS and similar agencies.
ECS was founded in 1965. Have its recommendations led to noticeable improvements in education? Has it publicly warned against any of the fads that have come and gone since ’65? And what about the worth of the advice coming from the other research and policy organizations that rely on public funding, i.e., the state education agencies, the regional education laboratories, the U.S. Department of Education, and other such groups?
In truth, none of these entities deal with the education community at arm’s length. Like stockbrokers who always say, “buy,” they rarely disagree with education’s mainstream about anything. Fads like whole language reading instruction, for example, have persisted for decades despite weak and contradictory research; yet the research and policy community has had little to say. And when individual researchers have expressed skepticism, they have found themselves out on a limb.
Policymakers need watchdogs that bark--and bark before policies are set and zillions spent.
The ECS response to the present study illustrates the problem. Despite a clear lack of supporting evidence, NBPTS has for years assured policymakers that its certificate represents exceptional teacher quality. A majority of states have accepted this assurance and enacted substantial bonuses for Board Certified teachers—all with the blessing of ECS.
When the Tennessee study found evidence that disagrees with the NBPTS claims, ECS took off its rose-colored glasses and got out its microscope—but not to examine the NBPTS advertising. Instead it commissioned a panel to critique the evidence that conflicts with NBPTS’s unsubstantiated claims!
A federally funded What Works Clearinghouse was recently created to undertake that which ECS and the others have chronically failed to accomplish--warn policymakers about poorly tested and wasteful innovations. Whether it too will be overly cozy with education’s hucksters, remains to be seen. However, if it turns out to be a good watchdog, policymakers might consider how much has been spent on the existing research and policy establishment and look for a way to redirect that funding. Effective oversight requires that fads be identified before they waste children’s lives and the taxpayers’ money.
J. E. Stone, Ed. D.
Education Consumers ClearingHouse
& Consultants Network
phone & fax 423-282-6832