Value-Added Assessment: An Accountability Revolution. In Marci Kanstoroom and Chester E. Finn, Jr. (Eds.), BetterTeachers, Better Schools. (Washington, DC: Thomas B. Fordham Foundation, 1999).

Citizens and policymakers concerned with the quality of public schooling are calling for improved standards and accountability.   They want objective evidence that students are meeting high achievement standards.  

Several states have responded by implementing periodic achievement examinations and setting a high score as the expected minimum.  Virginia, for example, set a requirement that 70% of each school's students must meet or exceed a challenging cutoff score on its new Standards of Learning test.  In a pilot run in 1998, less than 3% of schools were successful.  Similarly, Washington set a standard (but not a mandate) that 80% of each school's students should exceed a high minimum score on the Washington Assessment of Student Learning.  Only 70% of students at the State's highest performing school were successful. 

Inevitably such high rates of failure call the standards themselves into question (Sanders & Horn, 1995).  Can all schools reasonably be expected to reach the same standard and within the same time-frame?  For schools whose students are mostly high achievers, the task may be surmountable; but what about the schools with high percentages of disadvantaged students?  Moreover, can annual reports showing only the percentage of students reaching a set standard accurately and fairly reflect the work being done with low achievers?  Clearly, if percentage passing is the only indicator of school and teacher performance, the answer must be no.  A given school or teacher could make great strides with disadvantaged students yet fail to reach a standard that would easily be attained by a more advantaged population.

Beyond considerations of accuracy and fairness there is the question of whether a standards-based accountability policy creates perverse incentives.  Surely it does.   Teachers are encouraged to recommend difficult-to-teach students for programs in which they will be exempted from testing--special education, for example.  States such as Texas are already encountering this problem.  Also, instead of working to encourage all students to do their best, standards-based accountability effectively requires schools and school systems to put their greatest effort into ensuring that low achieving students reach the minimum.  In effect, they are encouraged to serve the least talented students at the expense of the most talented.  Few parents or policymakers would agree with these unannounced objectives.

If minimum score requirements are lowered in response to these concerns, a different question arises: Have policymakers accommodated standards to the least well equipped students and thereby set a mediocre expectation for all?  Alternatively, if they adjust expected minimums to account for differences in student populations, are they effectively consigning some students to inferior schooling?  Thus far states have responded to these questions by maintaining that students in all schools should be expected to reach the same high minimum standards.  Whether these standards will be politically sustainable in the face of disproportionate failure rates among disadvantaged and minority students remains to be seen.

Plainly, a better approach to school and teacher accountability is needed and fortunately one is available: Value-added assessment.  Value-added assessment permits citizens and policymakers to see how much the students in a given school or classroom are gaining annually in relation to their past history of achievement.  As explained below, it permits schools and teachers to be judged on the basis of how much progress they have made with their students regardless of entering achievement levels.  The perverse incentives resulting from standards-based accountability are thereby avoided. 

Yet value-added assessment does not replace standards-based assessment.  As a matter of sound policy, schools and school systems must be concerned both with student academic gains and with the percentage of students who attain the achievement minimums required for advancement or graduation.  What value-added assessment offers is a means whereby citizens, policymakers, and school administrators can accurately determine how much schools, school systems, and individual teachers are contributing to the attainment of expected achievement levels irrespective of the students they were assigned. 

Teacher Quality

There is widespread agreement that teacher quality is critical to classroom success.   A recent report by Sanders and Rivers (1997) has shown very substantial differences in the achievement gains earned by students who have excellent teachers versus those who have ineffective ones.  The differences were large enough to decisively shape the subsequent academic careers of the students in question. 

The teachers in the Sanders and Rivers study were all fully licensed as are most teachers in public schools.  Plainly, however, there were important differences in their effectiveness and to some extent these differences were due to their training.   Surprisingly, the question of the degree to which teacher training enables teachers to boost student achievement is one for which there are few clear answers.  The quality of teacher training has traditionally been assessed by a review of program inputs, e.g., whether the program includes certain courses, whether faculty are properly credentialed, whether the institution hosting the program is properly funded and accredited, etc.  Whether teachers trained by a fully approved program are able to do a superior job of producing measured student achievement, however, has not been unambiguously answered by research. 

In response to growing recognition of the need for improved teacher quality, the National Council for the Accreditation of Teacher Education (NCATE) has proposed changes in its standards for accreditation of teacher training programs.  Instead of the traditional review of inputs--i.e., a curriculum-based review--NCATE plans to conduct "performance-based" reviews--i.e., reviews of program strengths and weaknesses as evidenced by the demonstrated knowledge and skills of program graduates.   Performance-based reviews will purportedly answer the question of whether newly trained teachers are truly prepared to teach.  The problem, however, is that NCATE considers teachers to be well prepared when they understand, use, and believe in the pedagogical concepts that the teacher training community has been prescribing for years.   In all likelihood, improved teacher training by this definition will only produce more of the same.

NCATE is already the largest accreditor of teacher training programs.  Its standards are being promoted nationally as the key to improved teacher preparation.  If NCATE and its allies have their way, NCATE standards will become de facto national standards for teacher training. 

Value-added assessment provides a critically important alternative to NCATE's concept of teacher quality.  Instead of judging teacher quality by observing NCATE's preferred competence indicators, value-added assessment permits observers to appraise the record of student achievement gains produced by recent program graduates.  In other words, instead of judging teacher effectiveness only on the basis of indicators that NCATE believes predict effective teaching, value-added assessment permits effectiveness to be judged on the grounds of demonstrated success in producing actual achievement.  The Sanders and Rivers study noted above examined just such data drawn from the Tennessee Value-Added Assessment System (TVAAS). 

TVAAS is the heart of Tennessee's education accountability system.  TVAAS has been around since the late 1980s, and since 1995, it has been enlarged to produce value-added teacher effectiveness data for review by principals, and other school system personnel since 1995.  With a data base such as that provided by TVAAS, any state that desired to assess the quality of its teacher training programs could do so by aggregating the value-added performance of novice teachers trained by each program. 

Value-added assessment is further explained below and an example of a TVAAS "Teacher Report" is provided in the Appendix below. 

Value-added Assessment

Value-added assessment is a system of statistical analysis that summarizes annual gains in student achievement. The most recent and mathematically sophisticated version of value-added assessment was developed by Dr. William Sanders at the University of Tennessee (Sanders,Saxton, & Horn, 1997).  It has been used in Tennessee since the early nineties. A slightly different kind of value-added assessment using a different type of statistical analysis was used as early as 1984 in the Dallas (TX) Independent School District (Webster & Mendro, 1997). 

Tennessee tests all students annually in grades 3-8 with a customized version of McGraw-Hill's Terra Nova instrument.  The testing program is called the Tennessee Comprehensive Assessment Program (TCAP) and the results are used to inform students, parents, and teachers about individual pupil achievement.  The Tennessee Value-Added Assessment System (TVAAS) produces annual reports of the aggregate student achievement gains produced by each teacher, school, and system in Tennessee's public schools.   The annual reports for school systems are broken down by school and grade for each of the 5 subject areas measured by the TCAP exam (math, science, reading, language, and social studies).  The reports for teachers aggregate the gains earned by all students for which the teacher was responsible.  Under current Tennessee law, school and school system but not individual teacher reports are made public.

TVAAS reports express achievement gains in scale score points and in the form of comparisons to national, state, and local averages.  For example, 25 points is a typical amount of gain in student math achievement produced by 4th grade Tennessee teachers.  The average gain in math produced by 4th grade teachers nationally is 26 points.  Therefore, the typical Tennessee teacher is producing gains in 4th grade math equal to 96% of the national average.  In 4th grade science, by contrast, Tennessee teachers are producing gains equal to 115% of the national average. 

For critically important comparisons, a three-year rolling average is used to assure statistical stability.  For example, Washington County's Boones Creek Middle School produced a three year average gain (1993-95) of 65 scale score points in language arts for grades 5-8.  The national average gain in language arts for grades 5-8 is 50 scale score points.  Thus, Boones Creek Middle school produced gains equivalent to 130% of the national average language arts gains. 

By comparing student current achievement to their past performance and aggregating the results, value-added assessment  permits all stakeholders to see the impact of individual teachers, schools, and schools systems on the average progress of the students for which they are responsible. Not incidentally, value-added assessment can also be used by education decision-makers to assess the effectiveness of everything from the latest curricular innovations, to the preparedness of novice teachers, to the quality of the programs in which teachers were trained.

The statistical analysis employed in value-added assessment is an advanced form of "analysis of variance" called Henderson's "mixed model." It is described in "The Tennessee Value Added Assessment System" by Sanders, Saxton, and Horn (1997) and in several other sources cited below (Harville, 1976; Harville, 1977; Henderson, 1982; McLean, Sanders, & Stroup, 1991; Patterson, & Thompson, 1971; Raudenbush, & Bryk, 1988; Sanders, 1989).  It produces a "best linear unbiased estimate" of the influence on annual student achievement gains attributable to teachers, schools, and school systems. Of technical significance, value-added estimates of teacher influence are derived from a multi-year "layered" computational model and corrected by a "shrinkage estimate." These two features substantially reduce the possibility of false negative or false positive estimates and ensure that the resulting indicators of achievement gain are as exact as fairness will permit.

Compared to other methodologies for computing student achievement gains, value-added analysis is more precise and less vulnerable to manipulations that can distort results. For example, the hierarchical linear regression analysis used in Dallas, TX can exaggerate the differences attributable to a factor such as funding and correspondingly underestimate the differences attributable to teaching effectiveness if a variable such as per-pupil spending is prematurely entered into the statistical adjustment of student gains.

Preconditions

Value-added assessment is statistically robust but the validity of its results depends on certain preconditions. At a minimum, it requires annual testing of students in all grades with a reliable and valid achievement test. Portfolio assessment and other forms of assessment that lack reliability and objectivity will not suffice. Neither will standardized achievement tests that have been revised to enhance their marketability to educators at the expense of diminished academic content. No amount of analysis can transform the substance and meaning of fundamentally flawed data. Perhaps this limitation is best expressed in the statistician's time-honored adage "Garbage in, garbage out."

Yet, nothing in the use of value-added assessment precludes teachers from also using portfolios or any other form of assessment they deem necessary and desirable. Most educators believe that schooling should serve aims beyond those that can be measured by achievement tests and so they favor a variety of assessments. Parents and the public are not necessarily opposed to these broader aims, but they do disagree with the vast majority of educators about priorities. Whereas educators may view measured academic achievement as only one outcome among many, parents, taxpayers, and policymakers view it as the indispensable essential core of student (and teacher) results. No matter what other benefits good schooling may produce, those who fund the schools and who enroll their children in them will not be satisfied if the visible gains in objectively measured academic achievement are insufficient. Like an annual audit conducted by an external auditor, value-added assessment is an objective means whereby the consuming public can see whether its priorities are being respected and its hopes fulfilled.

In addition to requiring the use of a valid and reliable achievement test, value-added assessment requires that the items used in each annual testing be fresh, non-redundant, and tied to an underlying linear scale. The forms used at each grade level must include a sufficiently wide range of items such that "ceiling" and "floor" effects are highly improbable. Also the scores produced by the test must be reported on a common scale that spans the range of grades for which the test is appropriate.

The purpose of these requirements is to insure that the effectiveness of teachers, schools, and systems is tracked yearly, measured in understandable terms, and not artificially limited by the assessment process itself. In particular, the use of fresh test items insures that the gains calculated from value-added assessment represent student progress along the full spectrum of curricular objectives and not just improvements in the material sampled by the test, i.e., it discourages teaching to the test.

In order to insure fair assessments of teachers, Tennessee's value-added assessment reports include only those pupils who have attended school for at least 150 days and are not eligible for special education services. [Special education students are assessed through "individual education plans."] For teachers who have taught a given student for less than a full year, only those students who have been the teacher's responsibility for more than 75 days are counted.  Teachers whose subjects are not covered by the annual achievement examinations (e.g., art and music) are not assessed by value-added indicators.

Advantages

Value-added assessment offers several important advantages when compared to other forms of educational accountability.

1. It expresses teacher, school, and school system effectiveness in terms of increases in achievement over previous performance. In other words, in the computations that underlie teacher, school, and system effectiveness, each student is compared to his or her own record of achievement over a period of several years. By contrast, most present-day education accountability systems assess effectiveness by comparing current student achievement to an average or to an arbitrarily prescribed standard.  The failure of education accountability systems to consider gain relative to previous achievement can result in misleadingly negative evaluations for educators who are producing substantial but insufficient gains with disadvantaged students or misleadingly positive evaluations of educators who are producing mediocre gains with talented and advantaged students.

2. It excludes from the estimates of teacher, school, and school system effectiveness the influence of all pre-existing differences among students. These include but are not limited to race, socioeconomic status, previous learning, intelligence, and all other factors, known and unknown, that have influenced previous achievement. In contrast to "regression analysis" approaches to student gain assessment, the "mixed model" approach employs statistical "blocking" to remove the contribution of suspected biasing influences. Blocking has the advantage of removing differences without the necessity of measuring and computing the magnitude of each of the various excluded factors.

As counterintuitive as the notion of removing differences may seem, empirical studies of value-added assessment have demonstrated that it does remove differences among students and thereby levels the playing field for teachers, schools, and systems. Statistical analyses of Tennessee's value-added scores have shown no relationship between annual gains and previous achievement, race, eligibility for free or reduced lunch, or any other of a variety of potentially biasing differences among students (University of Tennessee Value-Added Research and Assessment Center, 1995).

Although value-added assessment removes pre-existing differences, it must be noted that such differences are not the only factors beyond an educator's control that can influence student gains.  Neither mixed-model analysis nor any other means of educational gain assessment automatically removes the effects that might result from "exogenous" influences arising during the course of the school year. For example, student illness or a natural disaster during the course of a school year might adversely affect someone's achievement gains.  Conversely, improvements in family income or the introduction of better community health care might contribute positively to achievement.

The influence of exogenous variables can and must be considered--especially as they impact a given school year--and mixed model methodology is able to incorporate such considerations. Happily, however, value-added analysis--properly interpreted--minimizes the need to do so. First, data is averaged over a period of years (permitting positive and negative influences to counterbalance each other) and, second, the gains of teachers, schools, and systems can be compared to the gains of other teachers, schools, and systems that have been exposed to the same or similar influences.

3. Mixed model value-added assessment is able to isolate the achievement effects produced by an individual teacher so long as students have been taught by that teacher for at least 75 days per semester. As a result, it is possible to assess teaching effectiveness regardless of whether teaching has been undertaken on a departmental basis, a team basis, or a traditional self-contained classroom basis.

4. The influence of a given teacher on student gains is expressed in the form of a "shrunken" or "regressed" estimate, i.e., an estimate that guards against an unfair assessment. In other words, the value-added system takes a very conservative approach to assessing teacher impact and thus insures that those who are identified as effective or ineffective are deserving of their classification.

5. Value-added assessment using mixed-model methodology makes use of all student scores despite the fact that some students will have missed tests and have incomplete sets of data. By contrast, methodologies such as regression analysis exclude students for whom complete data is lacking and thus they typically remove substantial numbers of students when analyses span 4 or 5 years.  Because poorer performing students are often the ones to miss tests, the exclusion of such students can substantially inflate achievement gain estimates.

6. As described above, value-added assessment permits comparisons to national average student gains and thus provides an understandable measure of student progress.   However, a caveat must be noted.  Gain scores depict how well students are progressing beyond their previous skills and knowledge but do not show how they stand with respect to an external benchmark of attainment, i.e., a national norm or prescribed standard. For this reason, comparison to national average gains is not a sufficient basis for judging education outcomes. A complete assessment requires consideration of both value-added performance and performance referenced to an external standard. Tennessee's value-added reports, for example, are concerned primarily with average student gains and the comparison of those gains to national averages.  Additionally, however, the TVAAS reports include average levels of achievement and appropriate national norms against which they may be judged.

An alternative to Tennessee's reporting system is one in which the annual learning gains produced by a given teacher, school, or system are compared to the annual learning gain necessary to bring students to an externally referenced benchmark. Although not currently used by any state, such a report would make it possible to consider both indicators simultaneously. For example, a school system with a substantial number of disadvantaged students might need to produce learning gains equal to 110% of the national average gains in order to reach national grade level standards by the 8th grade.

Conclusion

Although it employs some complex statistics, value-added assessment creates a simple but enormously important change in the educational landscape. It enables parents, taxpayers, and education decision-makers to see for themselves whether schools are working. It does so by greatly simplifying the process of interpreting reports on school effectiveness.

Such a change can revolutionize education. The public has been flooded with information about school quality but making sense of it has required experts and most of the experts have been educators who work for or with the schools. Now schools can produce a balance sheet and report an objective bottom line that is understandable to the interested citizen. Eventually, resources and students will flow to the effective schools and away from the ineffective ones.

References

For the interested reader there is a fairly extensive literature pertaining to value-added assessment. Although no one has yet written an account of mixed model methodology suitable for a general audience, it has been critically examined and reviewed by a number of scholars and policy experts.

Perhaps more importantly, value-added assessment has been used successfully in Tennessee for nearly 10 years and many of the interested parties have learned how to interpret and make use of it. Some schools have used it to identify weaknesses and have, as a result, made phenomenal gains.  Other schools--notably rural schools and schools with high percentages of economically disadvantaged students--have been able to show that they are doing a much better job of teaching than had been evidenced by indicators such as expenditures and the use of the latest educational practices. On the whole, student achievement in Tennessee has been improving over the years that value-added assessment has been in place.


For a general description see

McLean, R. A., & Sanders, W. L. (1984). Objective component of teacher evaluation A
     feasibility study (Working paper No. 199). Knoxville University of Tennessee, College of
     Business Administration.

Sanders, W. L., & Horn, S. P. (1994). The Tennessee Value-Added Assessment System
     (TVAAS) Mixed model methodology in educational assessment. Journal of Personnel
     Evaluation in Education, 8(1), 299-311.

Sanders, W. L. & Horn, S. P. (1995). Educational assessment reassessed The usefulness of
     standardized and alternative measures of student achievement as indicators for the
      assessment of educational outcomes. Education Policy Analysis Archives [online serial],
     3(6). Available http//epaa.asu.edu/epaa/v3n6.html

Sanders, W. L. & Rivers, J. C.  (1996, November)  Cumulative and residual effects of 
     teachers on future student academic achievement.    (Available from UTVARC , 225
     Morgan Hall, P.O. Box 1071, Knoxville, TN  37901-1071)

Sanders, W. L., Saxton, A. M., & Horn, S. P. (1997). The Tennessee value-added assessment
     system, a quantitative, outcomes-based approach to educational measurement. In Jason
     Millman (Ed.). Grading teachers, grading schools, Is student achievement a valid evaluation
      measure? (pp. 137-162). Thousand Oaks, CA Corwin Press.

Sanders, W. L., Saxton, A. M., Schneider, J. F., Dearden, B. L., Wright, S. P., & Horn, S. P.
     (1994). Effects of building change on indicators of student academic growth. Evaluation
      Perspectives, 4(1), 3, 7.

Webster, W. J. & Mendro, R. L.  (1997).  The Dallas value-added accountability system.   In
     Jason Millman (Ed.). Grading teachers, grading schools, Is student achievement a valid
     evaluation measure? (pp. 81-99). Thousand Oaks, CA Corwin Press.



For empirical findings with respect to value added model performance see

University of Tennessee Value-Added Research and Assessment Center. (1995). Graphical
      summary of educational findings from The Tennessee Value-Added Assessment System
     (TVAAS). (Available from UTVARC , 225 Morgan Hall, P.O. Box 1071, Knoxville, TN
     37901-1071)


For an evaluation and policy analysis see

Gormley, W. T. Jr., & Weimer, D. L. (1999). Organizational Report Cards. Cambridge, MA
     Harvard University Press.


For a comprehensive technical review see

Bock, R. D. & Wolfe, R. (1996, January 23). Audit and review of the Tennessee Value-Added
      Assessment System (TVAAS) Preliminary report. (Available from the Tennessee Office of
      Education Accountability, Comptroller of the Treasury, State Capitol, Nashville, TN 37243-
      0260)


For technical discussion of "mixed model" statistical analysis see

Harville, D. A. (1976). Extension of the Gauss-Markov Theorem to include the estimation of
      random effects. Annals of Statistics, 4(2), 384-395.

Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation and
      to related problems. Journal of the American Statistical Association, 72, 320-338.

Henderson, C. R. (1982). Analysis of variance in the mixed model Higher level,
      nonhomogeneous, and random regressions. Biometrics, 38, 623-640.

McLean, R. A., Sanders, W. L., & Stroup, W. W. (1991). A unified approach to mixed linear
      models. American Statistician, 45(1), 54-64.

Patterson, H. D., & Thompson, R. (1971). Recovery of interblock information when block sizes
      are unequal. Biometrika, 58, 545-554.

Raudenbush, S. W., & Bryk, A. S. (1988). Methodological advances in analyzing the effects of
      schools and classrooms on student learning. Review of Research in Education, 15, 423-479.

Sanders, W. L. (1989). A multivariate mixed model. In Applications of mixed models in
      agriculture and related disciplines (Southern Cooperative Series Bulletin No. 343, pp. 138-
      144). Baton Rouge Louisiana Agricultural Experiment Station.


For a review of several current approaches to assessing educational effectiveness

Millman, J. (Ed.). (1997). Grading teachers, grading schools. Is student achievement a valid
      evaluation measure? Thousand Oaks, CA Corwin Press, Inc.


For sample school and system value-added reports

Tennessee Department of Education
6th Floor, Andrew Johnson Tower
710 James Robertson Parkway
Nashville TN 37243-0375
(615)741-2731
http//www.state.tn.us/education/rptcrd98/rcstudent.htm


For school and system value-added reports in a consumer-friendly on-line display:

The Tennessean (Nashville)
http://www.tennessean.com/schools/


For technical information regarding value-added analysis and its implementation

Dr. William Sanders
Director
University of Tennessee Value-Added Research and Assessment Center
225 Morgan Hall
P. O. Box 1071
Knoxville, TN 37901
Phone (423) 974-7336
Fax (423) 974-7448
wsander2@utk.edu

APPENDIX

==========================================================================

1997 TVAAS TEACHER REPORT

*** TEACHER COPY ***

Teacher: XXXXXXXXXXXXXXXX  (000000000)
System: XXXXXXXXXXXXXXXX  (000)
School: XXXXXXXXXXXXXXXX  (000)

Grade: 4

Note: NDD = Not Detectably Different (within 2 standard errors).


Teacher 3-Year-Average Gain in Scale Score Units with Approximate 95% Confidence Intervals

         10                    20                   30                   40                     50
          +.........+.........+.........+.........+.........+.........+..........+..........+

Math                       (----L------T S N-------)

Reading                      (-S-N----LT---------)

Language     N             (--S----L-T---------)

Soc. St.             (----------LS*------------)

Science                      N    S(---------TL---------)



Legend:  T = Teacher Gain, L = System (LEA) Mean Gain, S = State Mean Gain, N = National Norm Gain.  An asterisk (*) indicates that 2 or more of the above symbols coincide.

The estimated teacher gains presented here are the official TVAAS estimates from the statistical mixed model methodology which protects each teacher from misleading results due to random occurrences.  Each teacher's gain is assumed to be equal to the average gain for the district until the weight of the data pulls the estimate away from the district average.

This year's estimates of previous years' gains may have changed as a result of incorporating the most recent student data.

==========================================================================



J. E. Stone is an educational psychologist and professor in
the College of Education at East Tennessee State University.
He also heads the Education Consumers ClearingHouse
(http//education-consumers.com).

This paper was completed with the support of The
Foundation Endowment, 611 Cameron Street, Alexandria, VA 22314
<tfe@laser.net>  (703) 683-1077 

Reprints may be obtained from The Foundation
Endowment or from the Thomas B. Fordham Foundation, 1627 K
Street, NW, Suite 600, Washington, DC 20006,
http://www.edexcellence.net   Publications: (888) 823-7474