BILL ANALYSIS
AB 1130
Page 1
Date of Hearing: April 29, 2009
ASSEMBLY COMMITTEE ON EDUCATION
Julia Brownley, Chair
AB 1130 (Solorio) - As Amended: April 22, 2009
SUBJECT : Academic performance
SUMMARY : States Legislative intent regarding the examination of
methods for making and reporting comparisons of school and
district academic achievement over time based on a cohort growth
measure. Specifically, this bill :
1)Makes findings and declarations regarding California's
accountability system and the benefits of incorporating a
cohort growth measure into that system.
2)States Legislative intent that the advisory committee advising
the Superintendent of Public Instruction (SPI) on matters
related to the API, to make recommendations to the SPI and the
State Board of Education (SBE) concerning establishment of a
methodology for measuring academic achievement by cohort to
more accurately measure academic growth for schools and
districts by providing the ability to determine:
a. High achievement with a growth rate indicating ability
to remain proficient or higher.
b. Low achievement with a growth rate indicating ability to
reach proficiency within a specified timeframe.
c. Low achievement with a growth rate indicating
significant inability to reach proficiency within a
specified timeframe.
3)States the intent of the Legislature that the advisory
committee take into consideration the pilot study conducted
pursuant to provision 10 of Item 6110-113-0890 of Section 2.00
of the Budget Act of 2007, federal statute and regulation
associated with the Elementary and Secondary Education Act
(the current reauthorization of which is known as NCLB)
accountability waivers granted by the U.S. Secretary of
Education, and measures in use in other states that reflect
student, subgroup, school and district growth.
AB 1130
Page 2
4)Requires that any measure of academic performance growth must
be in the public domain and meet specified statistical
standards, if it is:
a. Implemented by the SPI after being approved by the SBE.
b. Adopted by the state educational agency for the purposes
of federal education programs (i.e., the SBE), as part of
any plan or waiver request submitted to the federal
government under the Elementary and Secondary Education Act
(currently NCLB).
c. Adopted by the state as part of any other plan required
for receipt or allocation of federal funds.
EXISTING LAW :
1)Requires the SPI, with the approval of the SBE, to develop and
implement the API to measure the performance of schools, and
to include a variety of indicators, including achievement test
results, attendance rates, and graduation rates in that
measure.
2)Requires the SPI to establish an advisory committee to provide
advice on all appropriate matters relative to the creation of
the API.
3)Directs the advisory committee by July 1, 2005, to make
recommendations to the SPI on the appropriateness and
feasibility of a methodology for generating a measurement of
academic performance by using unique pupil identifiers and
annual academic achievement growth to provide a more accurate
measure of a school's growth over time.
FISCAL EFFECT : Unknown
COMMENTS : The SPI established, pursuant to SB 1 X1 (Alpert),
Chapter 3, Statutes of 1999-2000 First Extraordinary Session, an
advisory committee to advise the SPI and the SBE on all
appropriate matters relative to the creation of the API. SB 1
X1 also requires the SPI, with the approval of the SBE, to
develop the API to measure the performance of schools, and to
include a variety of indicators in that measure, including, but
not limited to, achievement test results, attendance rates, and
graduation rates. Currently only achievement test results are
AB 1130
Page 3
incorporated into the API, and the API is configured to produce
scores measuring a school's static performance at each grade
level, in each content area, in each year, at one point in time.
In addition the SPI also produces a "Growth API" that compares
this static performance from one year to the next by comparing
cohort or group scores. This growth API, however, does not
measure true value added for a specific group of students and is
not based on the year-to-year information for individual pupils;
in other words that measure may only be reflecting the
differences in cohorts of pupils that were in one grade level
over two different years, rather than actual growth for a fixed
set of students over time.
What is the impact of not being able to compare individual test
scores or the aggregate API over time? Even though individual
STAR test scores look the same from one year to the next and
allow a relative comparison to other students in the same grade
level in a given year, a student's scores are not comparable
across grade levels; this means that the student, parents, and
teachers can not tell if a student has improved or is achieving
at a lower level from one year to the next based on the test
scores that they receive. In short, we don't know whether the
520 that a student scores this year is higher, lower, or the
same as the 500 that student scored in the previous grade. The
primary impact of this shortcoming is that we are unable to
determine whether a specific instructional program designed to
maintain a student's academic growth or to accelerate that
student's growth is actually doing so. In the same way, the
inability to compare API results from one year to the next,
except through the current growth API that effectively measures
the results in one grade level for two successive and different
cohorts of students, restricts the state's ability to make
judgments about how a school's or district's instructional
program impacts its students' academic progress over time. In
other words, we are unable to tell whether school reform or
school improvement efforts are actually achieving results in
terms of academic growth in a school or district; even if we
could see growth, we are unable to really measure how great that
growth is. Clearly very large changes from one year to the next
would show up as very large changes in individual scores and in
the API, but large dramatic changes in one year are not
generally the result of school improvement. The lack of ability
to make comparisons over time has also hurt the state in terms
of its ability to take advantage of opportunities, provided as
part of the federal accountability system defined under No Child
AB 1130
Page 4
Left Behind, to adopt more flexibility in establishing how
schools and districts meet the standard of Adequate Yearly
Progress (AYP); this in turn has implications for schools and
districts moving into Program Improvement status and eventually
being mandated to accept various forms of state intervention,
including the possibility of state takeover.
Why can't we make these comparisons over time? There are three
primary obstacles that face any large-scale assessment and
accountability system that attempts to generate measures that
allow valid comparisons of achievement over time: cohort
instability, content discontinuity, and score incomparability.
Cohort instability simply refers to the fact that a school or
district won't have the same set of students in one grade this
year that it had in the previous grade the year before; in other
words, students move in and students move out of schools and
districts. This means that an aggregate measure, like the API,
is based on a different set of student scores in each of those
two years, and if the students from one year to the next are
different, then we can not know whether a change in the API
results from the work that the school, district, students and
parents have done or simply from the fact that the academic
achievement of the two sets of students is different. While
this problem may have an insignificant effect in some schools
and districts, California has schools and districts with
year-to-year turnover that exceeds 100 percent - meaning that
more students have left and come into the school or district
over the last year than were enrolled last year.
Content discontinuity refers to the fact that content upon which
scores and measures are based may not create a continuous
progression across all grade levels; the simplest examples of
this are in the California mathematics standards beginning at
grade 8 and in the English language arts standards beginning at
grade 9. The standards above those grade levels were developed
to recognize the variety of courses and course sequences that
exist across California middle and high schools, so the
standards exist more as a grade level block, rather than a
sequence of grade levels or content. In a school one student
may take a math sequence of algebra, geometry, second-year
algebra and pre-calculus, while another takes pre-algebra,
algebra, statistics and no math class; this content
discontinuity creates an oranges and apples problem that
complicates and possibly invalidates comparisons of aggregate
achievement across the grade levels for that school. This also
AB 1130
Page 5
creates a problem for comparing individual scores; for example
the student taking geometry and then second-year algebra sees
their test scores go up from one grade to the next, but if that
same student had taken second-year algebra first and then
geometry (as sequenced in some schools), that student's scores
would have gone down from one year to the next. In addition,
since the individual grade level tests in a given content area
can not, in the time allotted for testing, test all of the
content standards for that grade level and content area, there
is a sampling of content done for inclusion on the tests. So
even if the content standards were completely sequenced across
grade levels, the tests drawn from those standards still may not
reflect a continuous sequence of content. Any discontinuity in
content creates an oranges and apples problem such that growth
in achievement is not reflected in a student's scores across two
years - what would be reflected would simply be that student's
achievement on two different sets of content.
Score incomparability refers to how the underlying scores on the
tests are created. Even if content discontinuity were not at
issue, in order to compare an individual student's test scores
over time the scales on which the test scores are measured at
each grade level would have had to have been statistically
produced together for all of the grade levels so that there was
a progression of possible scores up the grades (one process for
producing a score scale that has this progression is referred to
as vertical scaling); other statistical mediation approaches
might also be used in order to make those scores comparable. As
an example, take two teachers who both grade their students'
tests on a scale of 0 to 100; can we say that a 90 on one
teacher's test is the same as a 90 on the other? Clearly not,
even if the test content were the same, because we know that
teachers grade differently and that their perceptions of what
gets a score of 90 may be different. However, if we took all of
the tests from both classrooms and examined the results, we
could produce a common scale that reflected the difference
between the two scores of 90 and every other score in the two
classes, and that allowed cross-class comparisons. This same
sort of statistical process would have to be used to allow
scores on a series of grade-level tests to be compared across
those grade levels. The scale scores on the tests in the STAR
program were developed independent of each other and thus do not
validly support this type of cross grade level comparison. Some
would argue that the cut-point or level setting process that is
used to establish the STAR performance levels (e.g., basic,
AB 1130
Page 6
proficient, advanced) mediates this shortcoming in the scale
scores, but the judgmental nature of such a standard setting
would require extensive statistical validation before it was
determined that this process supports comparisons over time. In
addition, the individual scores produced in the STAR program
form the basis for both the API and for measuring AYP; if the
underlying test scores do not support comparisons over time,
then these resulting aggregate measures will suffer from the
same problem.
How can the test scores and aggregate growth measures be made to
be comparable over time? There are many methodologies across a
broad spectrum of approaches that could be employed to either
eliminate or work around this problem. On one end of that
spectrum might be a full vertical scaling effort. In this
approach test questions from one grade level test would be
administered to students in adjacent grades and the results
would be used to create a common scale across the grade levels.
Thus a student's growth could be tracked as the student moves up
the common scale that runs from the lowest grade level up
through the highest scores at the highest grade level. This
approach is dependent upon the underlying content of the tests
being continuous; in other words movement on the common scale
has to reflect a progression through the content. It is possible
that applying this approach to California might mean a
re-examination of the content standards and test content in
order to ensure that this content continuity exists. Since the
API is an aggregation of STAR test scores, vertical scaling of
the test scores would eliminate most of the problems associated
with using the API to compare school and district performance
across time. At the other end of the spectrum might be
approaches that rely on statistical procedures to estimate or
project what score, on the average, should be achieved in a
given year based on the previous year's score or other
information. In this way a student's actual score can be
compared to the projected score, and a judgment could be made
about whether the student grew at a greater or lesser rate than
the average. This same sort of statistical mediation could be
used directly on an aggregate measure, such as the API, without
applying the approach to individual test scores.
There are also many other approaches and methodologies that
could be employed to allow comparisons over time. As with any
large-scale statistical procedure, the trade-off among these
procedures is generally between the increased validity and
AB 1130
Page 7
accuracy of the resulting measures and the comparisons that are
made using them, and the cost and time involved in implementing
that approach. At the two ends of the spectrum, a vertical
scaling process would be the most involved of the approaches,
while direct statistical mediations would be less costly and
faster. On the other hand statistical mediation does not solve
the underlying problems, but works around them; thus problems
such as content discontinuity would still exist and pose a
potential threat to the validity of the conclusions and
comparisons that we make with these test scores and
accountability measures.
This bill proposes to state Legislative intent to focus the
advisory committee on cohort growth, or the growth of aggregate
scores for a group of pupils as they move through grade levels.
According to the bill's sponsor, this approach would produce
estimates or projected aggregate scores that would be used to
determine whether actual aggregate growth was occurring at,
above or below some desired trajectory. In other words this
bill appears to propose a direct statistical mediation of the
aggregate accountability (API) measure based on a cohort
analysis. It is not clear whether this approach would be used
to generate individual pupil scores that would be comparable
over time.
According to the author, "AB 1130 is an important step toward a
reliable, more accurate and comprehensive measure of academic
achievement. It will also help inform the development of a
longitudinal growth model in California that will be used for
accountability and policy making purposes at the state,
district, and school level." The author also states that,
"Education leaders and researchers have found that including a
growth measure in accountability determinations produces a
catalyst to data driven decision making from the district to the
school site and even the classroom. In other states,
implementation of growth measures that are understandable to
teachers, parents and policy makers have fostered partnerships,
innovation and collaboration among public education
stakeholders."
There are two concerns with this bill. The first concern stems
from the lack of any specified timeline. Since there are no
deadlines, time references or time-related conditions that bear
on the delivery of the recommendations from the advisory
committee nor on the implementation of those recommendations by
AB 1130
Page 8
the SPI, the SBE, or the "state" - all of which are parties that
are seen by this bill as being authorized to trigger
implementation of the recommendations - there is no guarantee
that the recommendations are made by the advisory committee or
that those recommendations, if made, are implemented. The
second concern is also associated with implementation of the
recommendations. The bill does not require the advisory
committee recommendations to be implemented, but does refer to
the recommendations being implemented in any one of three ways:
a) by the SPI after being approved by the SBE (current law); b)
by the state educational agency designated for the purposes of
federal education programs (i.e., the SBE) as part of any plan
or waiver request submitted to the federal government under NCLB
or subsequent legislation replacing it; or c) by the state as
part of any other plan required for receipt or allocation of
federal funds. This ambiguity in specifying how and by whom the
changes envisioned in this bill would be implemented, does not
match the importance or the magnitude of these changes. Making a
change in how we measure progress of both students and schools
potentially has significant impacts on individual students,
schools and school districts in terms both the state and the
federal accountability system, as well as in overall school
reform; a change of this significance should have the
involvement of the Legislature and the Governor.
Provision 10 of Item 6110-113-0890 of section 2.00 of the Budget
Act of 2007 required the State Board of Education (SBE) and the
CDE to expand an existing study, examining academic growth
measures using existing longitudinal data of selected grades and
content areas, to evaluate multiple approaches for measuring
individual pupil annual growth on the state standards; the
Budget Act of 2007 also authorized the use of federal funds for
this purpose. The study was required to consider pupil cohorts
by selected grade level as well as pupil subgroups. The study
was required to provide:
1)Guidance on the utility of studied growth models to meet state
and federal accountability requirements.
2)Guidance on the ease of understanding and communicating the
meaning of studied growth measures to parents, educators,
policymakers, and pupils.
3)Potential cost impacts of the studied growth measures.
AB 1130
Page 9
4)Guidance on the use of studied growth measures in evaluating
individual pupil longitudinal data after the implementation of
the California Longitudinal Pupil Achievement Data System
(CALPADS).
The study, conducted by the Education Testing Service, examined
five different approaches to measuring growth, including
vertical scaling and different statistical mediations. The
study made recommendations that the state proceed with a
regression based approach, consider the development of vertical
scales, and not pursue certain specific statistical approaches;
the study also provided caveats about the problems involved in
these approaches, the possibility of misunderstanding or
misinterpretation of the resulting comparisons, and the
unintended consequences that could occur with the release of
growth information to students and parents. Problems with
misuse and misinterpretation, as well as unintended
consequences, present serious threats to the validity of any
approach used to produce measures of student or aggregate
achievement. This bill requires that the results of any adopted
or implemented growth model be in the public domain, be
replicable, and meet specified statistical standards related to
the accuracy (i.e., reliability) of the measure; the bill does
not establish a similar standard with respect to the validity of
the measure, or of the conclusions and comparisons made using
that measure.
Related legislation: This bill is one of four bills that propose
changes to the state's accountability system, specifically to
the API measure, and that will be heard by the Assembly
Education Committee this month. Those four bills are AB 173
(Price), AB 429 (Brownley), AB 1130 (Solorio), and AB 1435 (V.
Manuel Perez). The last page of this analysis provides a
side-by-side comparison of key features of these bills. AB 173
(Price), pending in the Assembly Education Committee, states the
intent of the Legislature to adopt a new measure to replace the
API, and requires the CDE to convene a new advisory board to
provide general guidance and make recommendations toward that
end. AB 429 (Brownley), pending in the Assembly Education
Committee, requires examination of methods for making and
reporting valid comparisons of individual academic performance
over time and for making potential improvements in the Academic
Performance Index (API), so as to be able to measure and report
both a student's and a school's academic growth over time. AB
1435 (V. M. Perez), pending in the Assembly Education Committee,
AB 1130
Page 10
requires the examination of assessment data related to the
acquisition of English language by English learners (EL) and of
EL proficiency with respect to making potential improvements in
the API.
Previous legislation: AB 2776 (Mullin), held in the Senate
Appropriations Committee in 2008, would have required
examination of the collection of individual student data, the
state's emerging data systems, the possibility of making real
comparisons of student performance over time, and the long-term
availability of assessment data related to the acquisition of
English language by English learners with respect to making
potential improvements in the API. AB 2478 (Huffman), held in
the Assembly Appropriations Committee in 2008, makes changes in
the issues on which the advisory committee advising the SPI on
the API is required to make recommendations. AB 519 (Mendoza)
would have required the incorporation of data regarding the
availability in high schools of a course of study that fulfills
University of California and California State University
admission requirements into the API, and the submission of a
plan for incorporating dropout data into the API. This bill was
later amended into different subject matter and author
(Committee on the Budget), and enacted as Chapter 757, Statutes
of 2008. SB 219 (Steinberg), Chapter 731, Statutes of 2007,
makes changes in the calculation of and in the process for
revising the API. AB 400 (Nunez), vetoed in 2007, would have
required the incorporation of additional measures of performance
into the API, including the rate at which pupils are offered a
course of study that fulfills University of California and
California State University admission requirements. AB 2167
(Arambula), Chapter 743, Statutes of 2006, establishes a
specific methodology for including graduation rates, as
previously required, in the API; also requires the SPI to report
annually to the Legislature on graduation and dropout rates in
the state. SB 1284 (Scott), held in the Assembly Appropriations
Committee in 2006, would have updated and made technical
amendments to statutes that establish the API. SB 1448
(Alpert), Chapter 233, Statutes of 2004, reauthorized the STAR
Program. SB 257 (Alpert), Chapter 782, Statutes of 2003,
requires the advisory committee established to advise the SPI on
the API to make recommendations to the SPI on a methodology for
generating a "gain" score measurement to provide more accurate
measure of a school's growth over time. AB 1295 (Thomson),
Chapter 887, Statutes of 2001, makes changes to the API to allow
small school districts to receive an API score, receive growth
AB 1130
Page 11
targets, and performance awards. SB 1 X1 (Alpert), Chapter 3,
Statutes of 1999-2000 First Extraordinary Session, known as the
Public Schools Accountability Act (PSAA), authorizes the state's
current accountability program, including establishment of the
PSAA Advisory Committee and development of the API. SB 2 X1
(O'Connell), Chapter 1, Statutes of 1999-2000, authorized
development of the high school exit examination, and established
a timeline for requiring passage of that examination in order to
qualify for the high school diploma. SB 376 (Alpert), Chapter
828, Statutes of 1997, authorized development and implementation
of the STAR Program.
REGISTERED SUPPORT / OPPOSITION :
Support
EdVoice (sponsor)
Opposition
None on file
Analysis Prepared by : Gerald Shelton / ED. / (916) 319-2087
AB 1130
Page 12
Comparisons of Current Law, AB 429, AB 1130, AB 1435, and AB 173 on
Key Elements in the Proposals to Improve California Assessment and
Accountability Measures
------------------------------------------------------------------------------------------
| | Current | AB 173 | AB 429 |AB 1130 (4/22/09 | AB 1435 |
| | Law | (4/14/09 | (introduced) | ver.) |(introduced) |
| | | ver.) | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Primary |Developed |Replace API |Facilitate |Facilitate |Add CELDT |
|proposal |API and |with new |growth |growth |and EL |
| |advises |measure |comparisons |comparisons |proficiency |
| |SPI on | | | |to API |
| |relevant | | | | |
| |matters | | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Improves |Created |Both with a |Both |Aggregate |Aggregate |
|individual or |aggregate |single |individual |accountability |accountabilit|
|aggregate |accountabil|measure |test scores |measure |y measure |
|measures? |ity | |and aggregate | | |
| |measure | |accountability | | |
| | | |measure | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Who makes |API |New advisory |API advisory |API advisory |API advisory |
|recommendations|advisory |board with |committee |committee |committee |
|? |committee |independent | | | |
| | |oversight | | | |
| | |consultant | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|Deadline for |July 1, |None - not |July 1, 2011 |None |July 1, 2010 |
|recommendations|2005 |implemented | | | |
|? | |until the | | | |
| | |Legislature | | | |
| | |appropriates | | | |
| | |federal funds | | | |
| | |for this | | | |
| | |purpose with | | | |
| | |DOF approval | | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
AB 1130
Page 13
|Recommendations|SPI |Not specified |SPI who |SPI and SBE |SPI |
| provided to | | |forwards to | | |
|whom? | | |SBE, | | |
| | | |Legislature, | | |
| | | |Dept of | | |
| | | |Finance | | |
|---------------+-----------+--------------+---------------+-----------------+-------------|
|How are |SPI may |Not specified |Upon |SPI may |SPI may |
|recommendations|implement | |Legislative |implement with |implement |
| implemented |with SBE | |action that |SBE approval, |with SBE |
|and when? |approval | |appropriates |SBE may |approval |
| | | |funds for this |implement as | |
| | | |purpose |part of NCLB | |
| | | | |plan, or state | |
| | | | |may as part of | |
| | | | |any other | |
| | | | |federal plan | |
| | | | |submitted | |
------------------------------------------------------------------------------------------