ECS Review of the Tennessee NBPTS Study Misses the Obvious
September 23, 2002
A Tennessee study of
16 teachers certified by the National Board for Professional Teaching
Standards (NBPTS) found that none were producing exceptional student
achievement gains. The Education
Commission of the States (ECS) appointed a panel headed by Penn Education dean
Susan Fuhrman to review it.
The
just-released review both misses the central point of the study and prompts
questions about the value of ECS policy recommendations.
The review focused primarily
on whether the study’s subjects were an adequate sample of all NBPTS teachers.
In fact, the 16 subjects
in the study were not treated as a sample. There was no attempt to generalize the findings to either the
Tennessee or the national population of NBPTS-certified teachers. Rather the 16 subjects were the entire
number of the NBPTS-certified teachers in Tennessee for whom value-added scores
were available, i.e., teachers in grades 3-8.
Precisely because the sample would have been too small, the study purposely
avoided any conclusions about the national NBPTS population.
Unlike studies that sample a
population and attempt to draw an inference about the larger group, the
Tennessee study was simply a multiple replication trial of the NBPTS certification
process. The value-added achievement gains
of 16 NBPTS-certified teachers were examined.
In 16 out of 16 cases they were found not to be exceptional
producers of student achievement.
When a certification process
is checked 16 times and found wrong every instance, any reasonable person would
say it isn’t trustworthy regardless of what might be inferred about others who
have been certified by the same process.
Here is an analogy: Suppose 100 football players are found to
weigh 300 pounds when weighed on a bathroom scale, and 16 of them are taken to
a doctor’s office and reweighed on a professional quality scale. If the 16 were found to weigh less than 250
on the doctor’s scale, the accuracy of the bathroom scale would be in doubt
regardless of what might be presumed about the 84 who had not yet been
reweighed.
Despite a four-month long
review, the Fuhrman panel overlooked the obvious. Instead of being commissioned to examine the evidence on both
sides of the NBPTS issue, it was given a narrow faultfinding assignment. As a result, it raised criticisms and
questions that are beside the point. It
is true that the Tennessee study does not answer a number of the panel’s
questions, but the reason is that access to Tennessee’s teacher-effect data is
tightly restricted and answers are mostly unavailable. In any case, Stone repeatedly offered to
answer any question had by the panel but was ignored.
Other observers have commented
favorably. For example, Stanford
researcher Eric Hanushek has said, “Stone’s study follows a well conceived
methodology.” The small number of
teachers in his study, “ . . . is not Stone’s fault or choice. It simply represents the available universe of
teachers.” “One thinks that, had
Stone’s study of 16 teachers supported the certification program, it would have
been widely publicized and little criticized.” (http://www.heartland.org/education/aug02/certified.htm)
Is ECS Helping
Policymakers?
The nature and circumstances
of the ECS review raises broader questions about the kind of advice that comes
from ECS and similar agencies.
ECS was founded in
1965. Have its recommendations led to
noticeable improvements in education?
Has it publicly warned against any of the fads that have come and gone
since ’65? And what about the worth of
the advice coming from the other research and policy organizations that rely on
public funding, i.e., the state education agencies, the regional education
laboratories, the U.S. Department of Education, and other such groups?
In truth, none of these
entities deal with the education community at arm’s length. Like stockbrokers who always say, “buy,”
they rarely disagree with education’s mainstream about anything. Fads like whole language reading
instruction, for example, have persisted for decades despite weak and
contradictory research; yet the research and policy community has had little to
say. And when individual researchers
have expressed skepticism, they have found themselves out on a limb.
Policymakers need watchdogs
that bark--and bark before policies are set and zillions spent.
The ECS response to the
present study illustrates the problem.
Despite a clear lack of supporting evidence, NBPTS has for years assured
policymakers that its certificate represents exceptional teacher quality. A majority of states have accepted this
assurance and enacted substantial bonuses for Board Certified teachers—all with
the blessing of ECS.
When the Tennessee study
found evidence that disagrees with the NBPTS claims, ECS took off its
rose-colored glasses and got out its microscope—but not to examine the
NBPTS advertising. Instead it commissioned
a panel to critique the evidence that conflicts with NBPTS’s unsubstantiated
claims!
A federally funded What Works
Clearinghouse was recently created to undertake that which ECS and the
others have chronically failed to accomplish--warn policymakers about poorly
tested and wasteful innovations. Whether
it too will be overly cozy with education’s hucksters, remains to be seen. However, if it turns out to be a good
watchdog, policymakers might consider how much has been spent on the existing
research and policy establishment and look for a way to redirect that
funding. Effective oversight requires that
fads be identified before they waste children’s lives and the taxpayers’
money.
J. E. Stone, Ed. D.
Education Consumers
ClearingHouse
& Consultants
Network
www.education-consumers.com
professor@education-consumers.com
phone & fax
423-282-6832