BILL ANALYSIS
AB 429
Page 1
Date of Hearing: April 22, 2009
ASSEMBLY COMMITTEE ON EDUCATION
Julia Brownley, Chair
AB 429 (Brownley) - As Introduced: February 23, 2009
SUBJECT : The Public Schools Accountability Act of 1999:
advisory committee
SUMMARY : Requires examination of methods for making and
reporting valid comparisons of individual academic performance
over time and for making potential improvements in the Academic
Performance Index (API), so as to be able to measure and report
both a student's and a school's academic growth over time.
Specifically, this bill :
1)Directs the advisory committee advising the Superintendent of
Public Instruction (SPI) on matters related to the API, to
make recommendations to the SPI by July 1, 2011, concerning
the establishment of a methodology for making the state's
assessment system longitudinally valid, and for measuring
academic growth more accurately and validly over time for
individual students and for schools.
2)Requires the advisory committee to use the pilot study
conducted pursuant to provision 10 of Item 6110-113-0890 of
Section 2.00 of the Budget Act of 2007 in making its
recommendations.
3)Requires the SPI to forward the committee's recommendations,
along with cost estimates and a timeline for implementation of
each recommendation, to the State Board of Education (SBE),
the appropriate policy and fiscal committees of the
Legislature, and the Department of Finance.
4)Prohibits these recommendations or any other proposal to
develop longitudinally valid measures from being implemented
until funds are appropriated by the Legislature specifically
for that purpose.
EXISTING LAW :
1)Requires the SPI, with the approval of the SBE, to develop and
implement the API to measure the performance of schools, and
to include a variety of indicators, including achievement test
AB 429
Page 2
results, attendance rates, and graduation rates in that
measure.
2)Requires the SPI to establish an advisory committee to provide
advice on all appropriate matters relative to the creation of
the API.
3)Directs the advisory committee by July 1, 2005, to make
recommendations to the SPI on the appropriateness and
feasibility of a methodology for generating a measurement of
academic performance by using unique pupil identifiers and
annual academic achievement growth to provide a more accurate
measure of a school's growth over time.
4)Establishes the Standardized Testing and Reporting (STAR)
Program to test academic skills in grades 2-11, and to report
individual and aggregate results.
FISCAL EFFECT : Unknown
COMMENTS : The SPI established, pursuant to SB 1 X1 (Alpert),
Chapter 3, Statutes of 1999-2000 First Extraordinary Session, an
advisory committee to advise the SPI and the SBE on all
appropriate matters relative to the creation of the API. SB 1
X1 also requires the SPI, with the approval of the SBE, to
develop the API to measure the performance of schools, and to
include a variety of indicators in that measure, including, but
not limited to, achievement test results, attendance rates, and
graduation rates. Currently only achievement test results are
incorporated into the API, and the API is configured to produce
scores measuring a school's static performance at each grade
level, in each content area, in each year, at one point in time.
In addition the SPI also produces a "Growth API" that compares
this static performance from one year to the next by comparing
cohort or group scores. This growth API, however, does not
measure true value added for a specific group of students and is
not based on the year-to-year information for individual pupils;
in other words that measure may only be reflecting the
differences in cohorts of pupils that were in one grade level
over two different years, rather than actual growth for a fixed
set of students over time.
What is the impact of not being able to compare individual test
scores or the aggregate API over time? Even though individual
STAR test scores look the same from one year to the next and
AB 429
Page 3
allow a relative comparison to other students in the same grade
level in a given year, a student's scores are not comparable
across grade levels; this means that the student, parents, and
teachers can not tell if a student has improved or is achieving
at a lower level from one year to the next based on the test
scores that they receive. In short, we don't know whether the
520 that a student scores this year is higher, lower, or the
same as the 500 that student scored in the previous grade. The
primary impact of this shortcoming is that we are unable to
determine whether a specific instructional program designed to
maintain a student's academic growth or to accelerate that
student's growth is actually doing so. In the same way, the
inability to compare API results from one year to the next,
except through the current growth API that effectively measures
the results in one grade level for two successive and different
cohorts of students, restricts the state's ability to make
judgments about how a school's or district's instructional
program impacts its students' academic progress over time. In
other words, we are unable to tell whether school reform or
school improvement efforts are actually achieving results in
terms of academic growth in a school or district; even if we
could see growth, we are unable to really measure how great that
growth is. Clearly very large changes from one year to the next
would show up as very large changes in individual scores and in
the API, but large dramatic changes in one year are not
generally the result of school improvement. The lack of ability
to make comparisons over time has also hurt the state in terms
of its ability to take advantage of opportunities, provided as
part of the federal accountability system defined under No Child
Left Behind, to adopt more flexibility in establishing how
schools and districts meet the standard of Adequate Yearly
Progress (AYP); this in turn has implications for schools and
districts moving into Program Improvement status and eventually
being mandated to accept various forms of state intervention,
including the possibility of state takeover.
Why can't we make these comparisons over time? There are three
primary obstacles that face any large-scale assessment and
accountability system that attempts to generate measures that
allow valid comparisons of achievement over time: cohort
instability, content discontinuity, and score incomparability.
Cohort instability simply refers to the fact that a school or
district won't have the same set of students in one grade this
year that it had in the previous grade the year before; in other
words, students move in and students move out of schools and
AB 429
Page 4
districts. This means that an aggregate measure, like the API,
is based on a different set of student scores in each of those
two years, and if the students from one year to the next are
different, then we can not know whether a change in the API
results from the work that the school, district, students and
parents have done or simply from the fact that the academic
achievement of the two sets of students is different. While
this problem may have an insignificant effect in some schools
and districts, California has schools and districts with
year-to-year turnover that exceeds 100 percent - meaning that
more students have left and come into the school or district
over the last year than were enrolled last year.
Content discontinuity refers to the fact that content upon which
scores and measures are based may not create a continuous
progression across all grade levels; the simplest examples of
this are in the California mathematics standards beginning at
grade 8 and in the English language arts standards beginning at
grade 9. The standards above those grade levels were developed
to recognize the variety of courses and course sequences that
exist across California middle and high schools, so the
standards exist more as a grade level block, rather than a
sequence of grade levels or content. In a school one student
may take a math sequence of algebra, geometry, second-year
algebra and pre-calculus, while another takes pre-algebra,
algebra, statistics and no math class; this content
discontinuity creates an oranges and apples problem that
complicates and possibly invalidates comparisons of aggregate
achievement across the grade levels for that school. This also
creates a problem for comparing individual scores; for example
the student taking geometry and then second-year algebra sees
their test scores go up from one grade to the next, but if that
same student had taken second-year algebra first and then
geometry (as sequenced in some schools), that student's scores
would have gone down from one year to the next. In addition,
since the individual grade level tests in a given content area
can not, in the time allotted for testing, test all of the
content standards for that grade level and content area, there
is a sampling of content done for inclusion on the tests. So
even if the content standards were completely sequenced across
grade levels, the tests drawn from those standards still may not
reflect a continuous sequence of content. Any discontinuity in
content creates an oranges and apples problem such that growth
in achievement is not reflected in a student's scores across two
years - what would be reflected would simply be that student's
AB 429
Page 5
achievement on two different sets of content.
Score incomparability refers to how the underlying scores on the
tests are created. Even if content discontinuity were not at
issue, in order to compare an individual student's test scores
over time the scales on which the test scores are measured at
each grade level would have had to have been statistically
produced together for all of the grade levels so that there was
a progression of possible scores up the grades (one process for
producing a score scale that has this progression is referred to
as vertical scaling); other statistical mediation approaches
might also be used in order to make those scores comparable. As
an example, take two teachers who both grade their students'
tests on a scale of 0 to 100; can we say that a 90 on one
teacher's test is the same as a 90 on the other? Clearly not,
even if the test content were the same, because we know that
teachers grade differently and that their perceptions of what
gets a score of 90 may be different. However, if we took all of
the tests from both classrooms and examined the results, we
could produce a common scale that reflected the difference
between the two scores of 90 and every other score in the two
classes, and that allowed cross-class comparisons. This same
sort of statistical process would have to be used to allow
scores on a series of grade-level tests to be compared across
those grade levels. The scale scores on the tests in the STAR
program were developed independent of each other and thus do not
validly support this type of cross grade level comparison. Some
would argue that the cut-point or level setting process that is
used to establish the STAR performance levels (e.g., basic,
proficient, advanced) mediates this shortcoming in the scale
scores, but the judgmental nature of such a standard setting
would require extensive statistical validation before it was
determined that this process supports comparisons over time. In
addition, the individual scores produced in the STAR program
form the basis for both the API and for measuring AYP; if the
underlying test scores do not support comparisons over time,
then these resulting aggregate measures will suffer from the
same problem.
How can the test scores and aggregate growth measures be made to
be comparable over time? There are many methodologies across a
broad spectrum of approaches that could be employed to either
eliminate or work around this problem. On one end of that
spectrum might be a full vertical scaling effort. In this
approach test questions from one grade level test would be
AB 429
Page 6
administered to students in adjacent grades and the results
would be used to create a common scale across the grade levels.
Thus a student's growth could be tracked as the student moves up
the common scale that runs from the lowest grade level up
through the highest scores at the highest grade level. This
approach is dependent upon the underlying content of the tests
being continuous; in other words movement on the common scale
has to reflect a progression through the content. It is possible
that applying this approach to California might mean a
re-examination of the content standards and test content in
order to ensure that this content continuity exists. Since the
API is an aggregation of STAR test scores, vertical scaling of
the test scores would eliminate most of the problems associated
with using the API to compare school and district performance
across time. At the other end of the spectrum might lay
approaches that rely on statistical procedures to estimate or
project what score, on the average, should be achieved in a
given year based on the previous year's score or other
information. In this way a student's actual score can be
compared to the projected score, and a judgment could be made
about whether the student grew at a greater or lesser rate than
the average. This same sort of statistical mediation could be
used directly on an aggregate measure, such as the API, without
applying the approach to individual test scores.
There are also many other approaches and methodologies that
could be employed to allow comparisons over time. As with any
large-scale statistical procedure, the trade-off among these
procedures is generally between the increased validity and
accuracy of the resulting measures and the comparisons that are
made using them, and the cost and time involved in implementing
that approach. At the two ends of the spectrum, a vertical
scaling process would be the most involved of the approaches,
while direct statistical mediations would be less costly and
faster. On the other hand statistical mediation does not solve
the underlying problems, but works around them; thus problems
such as content discontinuity would still exist and pose a
potential threat to the validity of the conclusions and
comparisons that we make with these test scores and
accountability measures.
This bill does not presume that any of these approaches is best
in terms of either maximizing the validity and accuracy of the
comparisons of individual scores or aggregate API measure that
will eventually be compared over time or in terms of minimizing
AB 429
Page 7
the costs of producing these comparable measures. Instead this
bill directs the advisory committee, with the expertise to
balance these goals, to make recommendations on the best course
for the state to proceed; the bill does, however, constrain the
advisory committee by requiring it to solve this lack of
longitudinal comparability for both individual assessment
results and for the state's aggregate accountability measure.
In other words, this bill leads the advisory committee to those
many possible approaches where individual test scores that can
validly be compared over time are developed and used to build up
to an API that is also longitudinally valid. What this approach
rules out is an approach that mediates the aggregate API measure
without allowing the underlying individual test scores to be
compared over time.
This bill also requires the SPI to forward the advisory
committee's recommendations, along with cost estimates and a
timeline for implementation of each recommendation, to the SBE,
the appropriate policy and fiscal committees of the Legislature,
and the Department of Finance; in addition, the bill prohibits
these recommendations or any other proposal to develop
longitudinally valid measures from being implemented until funds
are appropriated by the Legislature specifically for that
purpose. Making a change in how we measure progress of both
students and schools potentially has significant impacts on
individual students, schools and school districts in terms both
the state and the federal accountability system, as well as in
overall school reform; a change of this significance should have
the involvement of the Legislature and the Governor.
Despite establishing a deadline for the advisory committee to
make its recommendations to the SPI, those recommendations may
not be immediately implemented since this bill requires the
above mentioned Legislative action appropriating funds for this
purpose. Given the current fiscal situation, the Legislature
might be reticent about appropriating funds for this purpose,
however, the costs of implementing such a change in the state's
assessment and accountability system could clearly be borne by
federal funds, specifically those allocated to the state under
Title VI of the No Child Left Behind Act.
Provision 10 of Item 6110-113-0890 of section 2.00 of the Budget
Act of 2007 required the State Board of Education (SBE) and the
CDE to expand an existing study, examining academic growth
measures using existing longitudinal data of selected grades and
AB 429
Page 8
content areas, to evaluate multiple approaches for measuring
individual pupil annual growth on the state standards; the
Budget Act of 2007 also authorized the use of federal funds for
this purpose. The study was required to consider pupil cohorts
by selected grade level as well as pupil subgroups. The study
was required to provide:
1)Guidance on the utility of studied growth models to meet state
and federal accountability requirements.
2)Guidance on the ease of understanding and communicating the
meaning of studied growth measures to parents, educators,
policymakers, and pupils.
3)Potential cost impacts of the studied growth measures.
4)Guidance on the use of studied growth measures in evaluating
individual pupil longitudinal data after the implementation of
the California Longitudinal Pupil Achievement Data System
(CALPADS).
The study, conducted by the Education Testing Service, examined
five different approaches to measuring growth, including
vertical scaling and different statistical mediations. The
study made recommendations that the state proceed with a
regression based approach, consider the development of vertical
scales, and not pursue certain specific statistical approaches;
the study also provided caveats about the problems involved in
these approaches, the possibility of misunderstanding or
misinterpretation of the resulting comparisons, and the
unintended consequences that could occur with the release of
growth information to students and parents. Problems with
misuse and misinterpretation, as well as unintended
consequences, present serious threats to the validity of any
approach used to produce measures of student or aggregate
achievement. This bill requires the advisory committee to use
the findings of this study in making its recommendations for
developing a more accurate and valid measure of individual and
aggregate academic growth.
Committee amendments: This bill provides a deadline for the
advisory committee to make recommendations to the SPI, but does
not provide a subsequent deadline for the SPI to forward these
recommendations, along with cost estimates and a timeline for
implementation of each recommendation, to the SBE, the
AB 429
Page 9
appropriate policy and fiscal committees of the Legislature, and
the Department of Finance. Committee staff recommends, and the
author accepts, a Committee amendment to set a date of October
1, 2011, as the date by which the SPI is required to forward
this information.
Related legislation: This bill is one of four bills that propose
changes to the state's accountability system, specifically to
the API measure, and that will be heard by the Assembly
Education Committee this month. Those four bills are AB 173
(Price), AB 429 (Brownley), AB 1130 (Solorio), and AB 1435 (V.
Manuel Perez). The back page of this analysis provides a
side-by-side comparison of key features of these bills. AB 173
(Price), pending in the Assembly Education Committee, states the
intent of the Legislature to adopt a new measure to replace the
API, and requires the CDE to convene a new advisory board to
provide general guidance and make recommendations toward that
end. AB 1130 (Solorio), pending in the Assembly Education
Committee, requires examination of methods for making and
reporting comparisons of school and district academic
achievement over time based on a cohort growth measure. AB 1435
(V. M. Perez), pending in the Assembly Education Committee,
requires the examination of assessment data related to the
acquisition of English language by English learners (EL) and of
EL proficiency with respect to making potential improvements in
the API.
Previous legislation: AB 2776 (Mullin), held in the Senate
Appropriations Committee in 2008, would have required
examination of the collection of individual student data, the
state's emerging data systems, the possibility of making real
comparisons of student performance over time, and the long-term
availability of assessment data related to the acquisition of
English language by English learners with respect to making
potential improvements in the API. AB 2478 (Huffman), held in
the Assembly Appropriations Committee in 2008, makes changes in
the issues on which the advisory committee advising the SPI on
the API is required to make recommendations. AB 519 (Mendoza)
would have required the incorporation of data regarding the
availability in high schools of a course of study that fulfills
University of California and California State University
admission requirements into the API, and the submission of a
plan for incorporating dropout data into the API. This bill was
later amended into different subject matter and author
(Committee on the Budget), and enacted as Chapter 757, Statutes
AB 429
Page 10
of 2008. SB 219 (Steinberg), Chapter 731, Statutes of 2007,
makes changes in the calculation of and in the process for
revising the API. AB 400 (Nunez), vetoed in 2007, would have
required the incorporation of additional measures of performance
into the API, including the rate at which pupils are offered a
course of study that fulfills University of California and
California State University admission requirements. AB 2167
(Arambula), Chapter 743, Statutes of 2006, establishes a
specific methodology for including graduation rates, as
previously required, in the API; also requires the SPI to report
annually to the Legislature on graduation and dropout rates in
the state. SB 1284 (Scott), held in the Assembly Appropriations
Committee in 2006, would have updated and made technical
amendments to statutes that establish the API. SB 1448
(Alpert), Chapter 233, Statutes of 2004, reauthorized the STAR
Program. SB 257 (Alpert), Chapter 782, Statutes of 2003,
requires the advisory committee established to advise the SPI on
the API to make recommendations to the SPI on a methodology for
generating a "gain" score measurement to provide more accurate
measure of a school's growth over time. AB 1295 (Thomson),
Chapter 887, Statutes of 2001, makes changes to the API to allow
small school districts to receive an API score, receive growth
targets, and performance awards. SB 1 X1 (Alpert), Chapter 3,
Statutes of 1999-2000 First Extraordinary Session, known as the
Public Schools Accountability Act (PSAA), authorizes the state's
current accountability program, including establishment of the
PSAA Advisory Committee and development of the API. SB 2 X1
(O'Connell), Chapter 1, Statutes of 1999-2000, authorized
development of the high school exit examination, and established
a timeline for requiring passage of that examination in order to
qualify for the high school diploma. SB 376 (Alpert), Chapter
828, Statutes of 1997, authorized development and implementation
of the STAR Program.
REGISTERED SUPPORT / OPPOSITION :
Support
American Federation of State, County and Municipal Employees,
AFL-CIO
Association of California School Administrators (sponsor)
California Federation of Teachers
Californians Together
EdVoice
Los Angeles County Office of Education
AB 429
Page 11
San Francisco Unified School District
Santa Clara County Office of Education
Opposition
None on file
Analysis Prepared by : Gerald Shelton / ED. / (916) 319-2087
AB 429
Page 12
Comparisons of Current Law, AB 429, AB 1130, AB 1435, and AB 173 on
Key Elements in the Proposals to Improve California Assessment and
Accountability Measures
------------------------------------------------------------------------------------------
| | Current | AB 173 | AB 429 | AB 1130 | AB 1435 |
| | Law | | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Primary |Developed |Replace API |Facilitate |Facilitate |Add CELDT |
|proposal |API and |with new |growth |growth |and EL |
| |advises |measure |comparisons |comparisons |proficiency |
| |SPI on | | | |to API |
| |relevant | | | | |
| |matters | | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Improves |Created |Both with a |Both |Aggregate |Aggregate |
|individual or |aggregate |single |individual |accountability |accountabilit|
|aggregate |accountabil|measure |test scores |measure |y measure |
|measures? |ity | |and aggregate | | |
| |measure | |accountability | | |
| | | |measure | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Who makes |API |New advisory |API advisory |API advisory |API advisory |
|recommendations|advisory |board with |committee |committee |committee |
|? |committee |independent | | | |
| | |oversight | | | |
| | |consultant | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Deadline for |July 1, |None-advisory |July 1, 2011 |None |July 1, 2010 |
|recommendations|2005 |board not | | | |
|? | |implemented | | | |
| | |until the | | | |
| | |Legislature | | | |
| | |appropriates | | | |
| | |federal funds | | | |
| | |for this | | | |
| | |purpose | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Recommendations|SPI |Not specified |SPI who |SPI and SBE |SPI |
AB 429
Page 13
| provided to | | |forwards to | | |
|whom? | | |SBE, | | |
| | | |Legislature, | | |
| | | |Dept of | | |
| | | |Finance | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|How are |SPI may |Not specified |Upon |SPI may |SPI may |
|recommendations|implement | |Legislative |implement with |implement |
| implemented |with SBE | |action that |SBE approval, or |with SBE |
|and when? |approval | |appropriates |SBE may |approval |
| | | |funds for this |implement as | |
| | | |purpose |part of plan | |
| | | | |submitted to the | |
| | | | |federal | |
| | | | |government | |
------------------------------------------------------------------------------------------