Education Consumers Consultants Network

 

Response to Education Commission of the States Panel Report of September 23, 2002

 

*************************************************************************************************************************

 

The Controversy Regarding the NBPTS Briefing & Study

May 12, 2002

(updated October 5, 2002)

 

The Education Consumers ClearingHouse (ECC) is a paid subscription service--a Consumers Union for the consumers of public education.  Briefings on educational research are published as a service to ClearingHouse subscribers by the Education Consumers Consultants Network (ECCN)--a partnership of consultants affiliated with the ClearingHouse.  Occasionally they are publicized as a public service. 

 

The Briefing titled “Value-Added Achievement Gains of NBPTS-Certified Teachers in Tennessee:  A Brief Report” (Vol. 2, No. 5, May 2002, http://www.education-consumers.com/briefs/may2002.asp) reported that a study by J. E. Stone found that NBPTS-certified teachers in Tennessee were only average producers of student achievement gains as compared to other teachers in their school districts.  Stone is the principal partner of the ECCN.  Given persistent questions about the value of NBPTS certification (most recently:  Harvard Education Letter, http://www.edletter.org/current/abstracts.shtml - a4), policymakers were urged to suspend the program until definitive research can be undertaken. 

 

The Briefing and the study by Stone have drawn comments from several organizations:

 

Linda Seebach in the Rocky Mountain News, October 5, 2002 (NEW):

http://www.rockymountainnews.com/drmn/news_columnists/article/0,1299,DRMN_86_1459977,00.html

“No one need be surprised that the national board is digging in its heels to protect its very lucrative franchise. But why is the Education Commission of the States, supposedly an independent body, so eager to help out?”

Education Week, “ECS Review Discounts Study Critical of Teaching Board,” October 2, 2002 (NEW):

http://www.edweek.org/ew/ewstory.cfm?slug=05ecs.h22&keywords=Stone

“The review of Mr. Stone's seven-page study, which he unveiled last spring, does not charge that his conclusions are wrong, but rather that the methods that led to his conclusions were faulty.”

Thomas Fordham Foundation (NEW):  http://www.edexcellence.net/foundation/gadfly/issue.cfm?id=40#581

 

The Gadfly:  Study of NBPTS certification also scrutinized by a panel of experts

 

Even more big guns were brought out by the Education Commission of the States (ECS) to evaluate a small study that examined the effectiveness of teachers certified by the National Board for Professional Teaching Standards (NBPTS) in Tennessee. That study (actually a 4-page brief followed by 4 pages of data), by J.E. Stone of East Tennessee State University and the Education Consumers’ Clearinghouse, analyzed the value-added achievement gains produced by NBPTS-certified teachers in Tennessee, as generated by the much-vaunted Tennessee Value Added Assessment System. [For more about Stone’s study, see http://www.edexcellence.net/foundation/gadfly/issue.cfm?id=58#846.]

 

Stone found that none of the 16 Board-certified teachers who teach in grades 3-8 in Tennessee met a standard for exceptional teaching. (That standard was producing 115 percent of a year’s academic growth in their local school system in three core subjects over three years, a standard now used to identify exceptional teachers in a new incentive program in Chattanooga. [For more about the program, see http://www.edexcellence.net/gadfly/v02/gadfly12.html#reading1.])

 

ECS asked four prominent scholars to examine the Stone study: Dominic Brewer, Susan Fuhrman, Robert Linn, and Ana Maria Villegas. (We hope ECS will continue this admirable practice of asking independent scholars to review all future studies of the effectiveness of the NBPTS, not just short briefs produced by board critics.) The reviewers complained that it was unclear how the 16 teachers were selected for the study and expressed concern that the teachers included in the study might not be representative of Board-certified teachers in Tennessee, but Stone’s study makes clear that the 16 teachers are the only Board certified teachers in Tennessee who teach in grades 3-8. (Value-added scores are not available for Tennessee teachers in other grades.)

 

The reviewers’ main concern is that teacher value-added scores jump around from year to year and vary significantly by subject and school district. Volatility of gain scores does present a challenge for efforts to identify effective teachers this way. But Stone looks at both averages and scores from individual years in his analysis, and no matter what angle he used, Board-certified teachers simply didn’t produce exceptional gains in student learning. The Tennessee testing system is not perfect, and it would have been helpful to know more about the distribution of exemplary and deficient scores among the general teaching population in Tennessee, but it does not appear that Tennessee’s Board-certified teachers are setting records for the value they add to student achievement.

 

While none of the Board-certified teachers in Tennessee was able to meet the standard set by Chattanooga for exceptional teaching, at least four teachers of 4th or 5th grade in Chattanooga did meet that standard, according to Ken Jordan, a special assistant to Mayor Bob Corker. Those four teachers applied and were selected to teach in high-need elementary schools in the district after they submitted evidence that they had produced gains of at least 115 percent of one year’s growth in three core subjects over three years.

 

 

School Reform News (NEW):

 

Noted education researcher Eric Hanushek, a senior fellow at the Hoover Institution, told School Reform News that, contrary to the NBPTS’ insinuations, “Stone’s study follows a well-conceived methodology.”  He added that “Tennessee is not an aggressive NBPTS state,” and therefore Stone had no choice but to rely on a mere 16 teachers in the database.  “This weakness is not Stone’s fault, or his choice.  It simply represents the available universe of teachers.” 

Hanushek observed, “John Stone’s provocative study underscores one extremely important feature of U. S. education:  Widely acclaimed and expensive policies frequently escape any evaluation in terms of their true effectiveness.  Stone’s study is far from definitive, but it is [italics in the original] the evidence that is available. 

“The NBPTS certification process has become an important element of policy in several states, and large financial rewards flow to successful applicants.  But all of this happened without a thorough analysis of its effectiveness.” 

States and school districts have awarded pay increases or bonuses in the range of $5,000 to $7,500 per year for teachers winning national certification.  After paying the NBPTS a $2,300 application fee--a tab sometimes picked up by their districts--candidates prepare a portfolio of their work, videotape themselves teaching, and take an all-day examination. 

Hanushek said now that Stone has asked “the right question,” others--including states with a significant financial stake in the NBPTS process--should look at the evidence also. 

“One thinks that, had Stone’s study of 16 teachers supported the certification program, it would have been widely publicized and little criticized,” Hanushek commented.  (School Reform News, Vol. 6, No. 8, August 2002, http://www.heartland.org/education/aug02/certified.htm)

 

Comments from a number of other professors, researchers, and teachers are quoted.  They include:  Dr. George Cunningham of the University of Louisville, Dr. Louis Chandler of the University of Pittsburgh, John Tuepker, a teacher and AFT official in Mississippi, and Al Haskvitz, a national award winning teacher in California.  (School Reform News, Vol. 6, No. 8, August 2002, http://www.heartland.org/education/aug02/findings.htm)

 

Center for Education Reform: 

 

This shouldn't be a surprise. NBPTS certification is really just ordinary certification on steroids -- a puffed-up assessment of teachers' mastery of conventional certification standards. Considering the large body of research finding almost no correlation between certification and teacher effectiveness, it makes sense to expect no correlation between "super-certification" and student performance. What one should expect is proportionately inflated rhetoric about the value of the certification, which is exactly what one finds. 

(Education Reform Newswire, Vol. 4, No. 17, May 7, 2002, http://www.edreform.com/update/2002/020507.html).

 

Thomas Fordham Foundation: 

 

NBPTS-certified teachers flunk value-added test in Tennessee

(The Education Gadfly, Vol. 2, No. 19, May 9, 2002, http://www.edexcellence.net/gadfly/index.html).

Education Week:  (NEW)

To date, research on the value of the voluntary national credential—especially its effect on student achievement—has been thin.  (“Critical Study of NBPTS Spurs State Advisory Group to Act,” May 15, 2002). 

A news release by the NBPTS (http://www.nbpts.org) was critical of the Tennessee study and the Education Commission of the States (http://www.ecs.org/clearinghouse/35/57/3557.htm) called for a review by a “panel of unbiased, distinguished educators and researchers.” 

 National Board for Professional Teaching Standards news release of May 7, 2002:   

 

The recent report by J. E. Stone about testing results in Tennessee on National Board Certified Teachers is hardly independent research.  Stone has been a frequent opponent of the National Board and by his own admission supports marketplace models for teaching.  He opposes certificates, licensure, and credentialing and calls those efforts "meaningless."

In this review of test scores, Stone uses 16 out of 40 National Board Certified Teachers (NBCTs) in a state with one of the lowest percentages of NBCTs (.07%).  The representation is a minuscule percentage as pointed out by Education Daily and represents less than .01 of total NBCTs.  Mr. Stone is well aware of the fact that a significant study of the impact of NBCTs on student achievement is currently underway by William Sanders who developed the "value added" system in Tennessee.  Mr. Sanders did not approve or authorize Stone's "so called" research model.  A cursory look at the information used for the 16 teachers suggests there is not enough data to make good judgements about the effects of NBCTs.

The National Board is looking forward to the results of the upcoming study that we have commissioned from Sanders that focuses on a much larger sample of NBCTs in the state of North Carolina.  This study will compare more than 800 teachers; those who achieved National Board Certification, those who went through the process and did not achieve certification, and teachers who have not been through the process.

We also welcome the review of the Stone study by the Education Commission of the States (ECS).  ECS is assembling a panel of "unbiased, distinguished educators and researchers" to conduct an independent review of the study's procedures, data and findings, and issue a public report of its conclusions. 

[The remainder of the release is available at:  http://www.nbpts.org]

 

Reply and Comment

 

Plainly, the Education Consumers ClearingHouse (ECC), its Consultants Network (ECCN), and the present study represent an independent viewpoint.  For a consumer organization or the studies it commends to be otherwise would be the proverbial “kiss of death.”  Without independence, the ClearingHouse would be out of business.   

 

The ClearingHouse would not expect the NBPTS to endorse this study.  Manufacturers and vendors, for example, frequently do not agree with the assessments of their products by consumer organizations.  The ECC’s stock-in-trade is credibility with the consumer, not acceptance by the education community. 

 

The NBPTS objects to the Tennessee study on the grounds that its author (Stone) has been a frequent critic of NBPTS; that he supports "marketplace" models for teaching; and that he characterizes teaching certificates, licensure, and credentials as meaningless. 

 

These objections are factual but relevant only from the standpoint of an unreasonably stringent standard of scholarly impartiality.  Scholars rarely study a subject about which they have no opinion.  The question is whether their opinion shaped the outcome.  In any case, it is apparent that NBPTS and ECS have not applied this standard to the several studies of NBPTS that have reported supportive findings. 

 

As to a “marketplace” point of view, in fact, the ClearingHouse and its Consultants Network are founded on the belief that the educational priorities of the parties who furnish the children and the money should be respected by the public servants whose salaries they pay. 

 

It is also true that Stone, Cunningham, and Crawford (2001, http://www.education-consumers.com/consultants/opinion3.pdf) have criticized NBPTS, NCATE, and INTASC standards on the grounds that they urge teaching practices unsuited to the attainment of student achievement—an assertion vindicated by the Tennessee evidence.

 

Finally, in a statement supporting the recent Abell Foundation report (http://www.education-consumers.com/articles/teacher_certification.asp), the ECCN including Stone have argued that existing teaching certificates, credentials, and licensure are meaningless with regard to the public’s interest in protection from unsound and ineffective teaching.

 

Stone has remarked, "Consider the fads to which public school students historically have been subjected by fully credentialed and licensed educators.  Physicians who use questionable practices are sued and called quacks.  Engineers whose bridges collapse are fined and lose their licenses.  Educators who institute fads are called innovators and given more funding."

 

The Size of the Tennessee Study

 

The NBPTS's chief complaint about the Tennessee study was that the number of teachers (16) was small and therefore may not be representative of NBPTS-certified teachers in general (a point carefully discussed by Stone's report).  

 

Stone's reply:  "While it is statistically possible that the 16 teachers available in TN just happen to be the only average performers in the NBPTS-certified population, don't bet the farm on it.  Surely Tennessee isn't so unlucky that it got them all." 

 

And what if there were only 16 mediocre NBPTS-certified teachers out of 16,000?  If 16 certified lifeguards were found to be mediocre swimmers, the finding would not be dismissed as statistically insignificant. 

 

Like an IQ score, NBPTS certification is a clinical statement.  NBPTS is certifying that each teacher is qualified, not that the NBPTS-certified group has an acceptable average. 

 

NBPTS’s and ECS’s Concerns and Priorities

 

If the NBPTS’s top priority were ensuring that their teachers are as skilled as advertised, they would take careful note of the Tennessee study and consider whether they should suspend their promotional campaign until fuller data is available.  Instead, they seem intent on discrediting it and defending the status quo. 

 

In truth, if policymakers continue to presume the validity of NBPTS certification and commit billions more to NBPTS teacher bonuses, it may become politically impossible to abandon the program regardless of what future studies show. 

 

This scenario is reminiscent of the life cycles had by the various teacher career-ladder plans that were adopted in the early nineteen eighties.  They too involved complex teacher assessment schemes that were similarly unsupported by proof of teacher effectiveness.  In Tennessee, for example, all that remains of its career ladder are the teacher bonuses and tax increases. Policymakers who are considering adoption of NBPTS bonuses would be well advised to consider Tennessee's experience and its budgetary consequences. 

 

The same might be said of the response made by the Education Commission of the States (ECS)  (http://www.ecs.org/clearinghouse/35/57/3557.htm).  Instead of alerting its members and noting that NBPTS certification has never been clearly linked to student achievement, ECS took the unprecedented step of calling for the Tennessee study to be reviewed by an 'unbiased, distinguished' review panel.  By contrast, ECS failed to express skepticism about any of the several NBPTS sponsored studies of the past 10 or so years, some of which included as few as 3 NBPTS-certified teachers [see J. Hattie, J. Clinton, M. Thompson, & H. Schmidt-Davis, Identifying Highly Accomplished teachers: A validation study, A Research Report of the National Board for Professional Teaching Standards Technical Analysis Group (Greensboro, North Carolina: Center for Educational Research and Evaluation, University of North Carolina-Greensboro, 1995)]. 

 

ECS has played a valuable role in educational reform but it may have a blind spot when it comes to NBPTS.  Governor James Hunt--a former ECS chair--was the founding chair of NBPTS and Ted Sanders--current president of ECS--is one of the founding commissioners of the NBPTS's greatest supporter, the National Commission on Teaching and America's Future.  Like the state departments of education of its constituency, ECS tends to represent the education community’s prevailing views.  In fact, NBPTS’s early endorsement by ECS (see ERIC ED 304444-1988, http://www.ed.gov/databases/ERIC_Digests/ed304444.html) is emblematic of its mainstream acceptance.

 

Unless ECS President Ted Sanders is confident that the forthcoming studies by Dr. William Sanders and others will not suggest major revisions in NBPTS, we would urge that he examine all of the NBPTS research and caution the states that the lack of supporting data may mean that they are committing millions to unwarranted teacher bonuses.  Historically educational fads have been “discovered” only after the money has been spent.  Perhaps the NBPTS initiative could become the one exception.