BILL ANALYSIS �
SB 547
Page 1
Date of Hearing: July 6, 2011
ASSEMBLY COMMITTEE ON EDUCATION
Julia Brownley, Chair
SB 547 (Steinberg) - As Amended: June 13, 2011
SENATE VOTE : Vote not relevant
SUBJECT : Public school performance accountability
SUMMARY : Replaces the state's current measure of school and
school district academic performance, the Academic Performance
Index (API), with the Education Quality Index (EQI), an index of
school and school district quality that is comprised of multiple
indices, each reflecting a different dimension of school or
district performance. Specifically, this bill :
1)Makes findings and declarations as to the challenges faced by
and the needs of California's pupils.
2)Makes findings and declarations regarding the state's current
accountability system and the opportunity presented to
transition to a new approach.
3)States Legislative intent that the API evolve to encompass
other valuable metrics in addition to test scores, graduation
rates and dropout rates, and that a more comprehensive set of
expectations and aspirations for California's public schools
be reflected in the state's school accountability system.
4)Makes current law implementing the API inoperative on July 1,
2013 and repealed as of January 1, 2014.
5)Requires the Superintendent of Public Instruction (SPI), in
consultation with the current Public School Accountability Act
(PSAA) advisory committee, to develop the EQI system for
schools and school districts, and requires the State Board of
Education (SBE) to adopt the EQI no later than August 1, 2013,
and after providing for public input.
6)Requires, commencing with the 2013-14 school year, all schools
and school districts to be evaluated using the EQI.
7)States Legislative intent that the EQI provide a comprehensive
and transparent measurement of pupil performance and school
SB 547
Page 2
quality to better inform parents, pupils, teachers, school
administrators, policymakers, and the public about public
school performance using multiple indicators of pupil, school,
and school district quality and performance.
8)Requires the EQI to be to be developed for each school type
and school district, and to include, but not be limited to,
the following:
a) For schools and school districts maintaining any of
grades 9 through 12, the:
i) State Assessment Index (SAI).
ii) Graduation Rate Index (GRI).
iii) College Preparedness Index (CPI).
iv) Career Readiness Index (CRI).
b) For schools and school districts maintaining grade 8,
the:
i) SAI.
ii) GRI.
iii) A valid and reliable measure or measures of pupil
access to and performance in college and career
preparatory and exploratory experiences is to be
considered, if maintained in that school.
c) For schools and school districts maintaining
kindergarten or any of grades 1 to 7, inclusive, the SAI.
9)Requires the SPI, in consultation with the PSAA advisory
committee, to develop and recommend to the SBE for adoption:
a) The component indices listed in 8) a) above for school
districts and for each school type, including alternative
schools, as specified.
b) The relative weights of the component indices, and a
total value for the EQI, consistent with the following
requirements:
i) For schools and school districts maintaining
kindergarten and any of grades 1 through 8, the SAI is
required to comprise no less than 40 percent of the value
of the EQI.
ii) For schools and school districts maintaining any of
grades 9 through 12, the SAI is required to comprise no
more than 40 percent of the value of the EQI.
iii) Within the EQI, the weights assigned to the CPI and
CRI are required to be equal.
c) No less than one additional component index for schools
maintaining kindergarten or any of grades 1 through 7.
d) An annual ranking or evaluation system for the EQI.
SB 547
Page 3
e) An annual growth target for the SAI and the GRI,
including targets for numerically significant pupil
subgroups, defined to be subgroups with more than 50 pupils
with valid test scores.
f) Whether a growth target or an annual measurement of the
narrowing of the achievement gap should be established for
each additional component index, including targets for
numerically significant pupil subgroups.
g) The relevant indices and indicators necessary to meet
and comply with federal law.
h) Any additional data elements, and connections between
existing data systems, that are identified as needed, after
consulting with the state's three higher education
segments, the Employment Development Department, and other
appropriate entities.
10)Requires the SPI to annually release results and supporting
information on the EQI and its component indices.
11)Requires the SPI, in consultation with the PSAA advisory
committee and when additional valid and reliable data become
available, to develop and recommend to the SBE additional
indices, as specified, and to take into account the
appropriateness of particular indices and indicators to grade
spans and school types; also prohibits any additional
component index from being included in the EQI until one full
school year after its adoption by the SBE.
12)Requires the SPI, no later than July 1, 2014, to report to
the Governor and the Legislature on the development,
implementation and impacts of the EQI and its component
indices, and requires the SPI, in consultation with the PSAA
advisory committee and no later than July 1, 2018, to report
to the Governor and the Legislature on the effectiveness and
reliability of the EQI and any statutory changes needed for
improvement.
13)Specifies that the SPI not be limited in the development of
the EQI by the scope of the California Longitudinal Pupil
Achievement Data System (CALPADS) or any other relevant data
system.
14)Authorizes the SPI to develop and recommend to the SBE for
adoption any regulations necessary to implement these
provisions.
SB 547
Page 4
15)Requires the SAI to be comprised of pupil scores from the
state's current standards-based achievement tests or any valid
and reliable successor assessments adopted by the SBE, and the
high school exit examination; also specifies inclusion and
exclusion rules, disaggregation requirements, and reporting
requirements and authorities.
16)Requires the GRI to be comprised, as appropriate to the grade
configuration of the school or school district, of:
a) Four-year, five-year, and six-year graduation rates,
that are appropriate to school types and where schools and
districts are required to be granted full value for
graduating, in five or more years, pupils with
disabilities.
b) Rates, as determined to be appropriate, at which pupils
successfully promote from one grade to the next.
17)Requires the CPI to be comprised of multiple valid, reliable,
and stable measures of pupil preparedness for postsecondary
education, that may include, but is not limited to district,
school and subgroup rates reflecting completion of:
a) Courses that fulfill the admission requirements for
California's four-year public universities.
b) College preparedness assessments, such as the Early
Assessment Program (EAP), the Preliminary SAT (PSAT), the
SAT and the ACT, as administered to high school pupils, and
rates at which pupils who complete these assessments can
demonstrate they do not need remedial coursework in order
to perform college-level work.
c) College preparatory and accelerated learning
opportunities, at a satisfactory level.
18)Requires the CRI to be comprised of multiple valid, reliable,
and stable measures of pupil readiness for career, including,
but not limited to:
a) Rates at which pupils satisfactorily complete a
designated career pathway.
b) Rates at which high school pupils earn an industry
validated certificate, license, or the equivalent in the
designated occupation for which it is issued, or rates at
which pupils are on track to earn such.
c) The performance of pupils on assessments designed to
assess the degree to which pupils have acquired the skills
and knowledge necessary to be successful in a specified
SB 547
Page 5
occupation or in the general workforce.
d) Rates at which pupils move on to additional training or
employment, or generate earnings related to that training
and employment.
EXISTING LAW requires:
1)The SPI to develop the API to measure the performance of
schools, and to include a variety of indicators in that
measure, including, but not limited to, achievement test
results, attendance rates, and graduation rates.
2)School districts to offer to all otherwise qualified pupils in
seventh through twelfth grades both of the following:
a) A course of study fulfilling the requirements and
prerequisites for admission to California's public
institutions of postsecondary education.
b) A course of study that provides an opportunity to attain
entry-level employment skills in business or industry upon
graduation from high school.
FISCAL EFFECT : Unknown
COMMENTS : The SPI established, pursuant to SB 1 X1 (Alpert),
Chapter 3, Statutes of 1999-2000 First Extraordinary Session,
the PSAA advisory committee to advise the SPI and the SBE on all
appropriate matters relative to the creation of the API. SB 1
X1 also requires the SPI, with the approval of the SBE, to
develop the API to measure the performance of schools for
reporting and accountability purposes, and to include a variety
of indicators in that measure, including, but not limited to,
achievement test results, attendance rates, and graduation
rates. Currently only achievement test results are incorporated
into the API, and the API is configured to produce scores
measuring a school's static performance at each grade level, in
each content area, in each year, at one point in time.
Having an API that focuses solely on achievement test results
may be too narrow in that it does not reflect information about
other student outcomes (e.g., dropout and graduation rates,
college readiness, preparation for the workplace) that is
important in measuring the performance of districts, schools and
subgroups. In addition, focusing solely on test scores may
SB 547
Page 6
actually lead to conclusions that are incorrect; as a perverse
example, a school that sees an increase in the number of
students dropping out could easily see a resulting increase in
test scores, and thus in the API for that school or district (if
the students dropping out tended to have below average test
scores), yet most would agree that this increase in test scores
and API are not reflective of an improvement in performance or
the quality of education in that school or district. The
solution to this problem would be to broaden the set of measures
that are composited to form the accountability measure. The
Legislature foresaw this when it initially authorized the
development of the API to be an index (i.e., a composite number
reflecting a number of component measures) based on data from
multiple measures, including achievement test results,
attendance rates, and graduation rates.
Opponents of including anything other than results on the
state's achievement test results in the API argue that including
data on other assessments, dropouts, graduation rates, and other
non-testing dimensions of educational performance will both
dilute the meaning of the API and skew its emphasis toward high
schools, resulting in the focusing of more resources at that
level to the detriment of elementary and middle schools. The
dilution argument assumes that state achievement test results
incorporate and reflect all aspects of school performance, or at
least the only important aspect; the increasing incidence of
high achieving pupils dropping out of high school is a counter
example to the claim that test scores alone show how well a
school is serving its pupils. In addition, since the
Legislature can target the use of resources by grade span as
well as by API ranking, this effect, to the extent that it would
exist, could be easily moderated.
Currently, California's public school accountability system has
no mechanism for measuring success in terms of outcomes or
opportunities beyond performance on core academic assessments,
including civic and community responsibility, career or college
readiness and acquisition of life skills; the current
accountability system also has no way of identifying or
rewarding those schools that educate the whole pupil in any way
other than is measured by testing skills and knowledge in
mathematics, language arts, history and science.
The indices proposed in this bill are intended to replace and
augment the information composited in the current API, thus
SB 547
Page 7
including information on student opportunities and student
success so as to provide broader and more comprehensive
information on school performance than is available from the API
focus on achievement test scores alone. The proposed EQI would
aggregate information from at least four sub-indices, each
related to different dimensions of student performance and
opportunity, including academic performance, advancement and
graduation rates, preparation for career or post-secondary
education, and enhanced student learning opportunities.
Academic performance, in the form of STAR test results in a
limited number of subject areas, are included in the API;
attendance and graduation rates are authorized to be included in
the API by the SPI, but have never been so included. The
remaining areas of performance or opportunity are not authorized
to be included in the API, although the state does produce
various individual indicators related to some of these
dimensions.
According to the author, "The existing Academic Performance
Index (API) has encouraged schools/districts to focus so hard on
test scores that other important aspects of education have been
deemphasized. Career technical education, arts and music, work
experience and other important learning has been marginalized in
favor of coursework that improves scores in tested subjects.
Creativity, communication skills, the ability to work in teams,
soft skills (in addition to academic proficiency) - all are
important aspects of a well-rounded education, and our system of
public school accountability needs to better reflect that." The
author also indicates that incorporating college and career
readiness indicators into accountability systems for high
schools is not a new idea, in that it has been used in other
states, including Florida, which began holding high schools
accountable for participation and performance in Advanced
Placement courses or completing other college-level work while
still in high school, and Colorado, which in 2009 incorporated
postsecondary and workforce readiness into its school ranking
system.
The author summarizes the motivation for this bill by stating
that, " It is time for the API to evolve to what it was
originally intended to be: an index that encompasses other
valuable indicators in addition to student test scores. These
additional indicators should include rates of graduation and
promotion from one grade to the next; rates of student readiness
for college and career; and measures of student engagement and
SB 547
Page 8
achievement in subjects areas outside the academic core
curriculum, including the arts."
The proposal in this bill for a comprehensive expansion of the
accountability measure that are used to provide public
information on and make state-level accountability judgments
about the state's K-12 public schools, provides the clear
benefit of basing that information on and judgments about
schools on a broader base of multiple measures of the
characteristics that define school success or failure. The lack
of this broad base of multiple measures has long been a
criticism of the API. However, this proposal is not without its
problems, most of which could be overcome, or else ignored as
part of the tradeoff for better accountability information;
these problems include:
1)The bill provides for a discrete transition from the API to
the EQI system, with no overlapping period when both measures
are being produced. This means, unless the SAI exactly
reflects the current composition of the API, that there will
be no public comparability between the API accountability
information that is produced for the 2012-13 fiscal year and
that produced under the EQI system for the 2013-14 fiscal
year. Certainly enough technical information on the API is
available to allow the research community to produce 2013-14
APIs for comparison purposes, but the Legislature may wish to
consider whether a one-year overlap in reporting for the two
measures would enhance public information and further the
understanding of the proposed new accountability measures.
2)Descriptive statistics, such as rates, averages and indices,
are commonly used in public policy analysis and in many other
situations where there is a large amount of raw data that
would be difficult for any individual to process and
understand; descriptive statistics typically combine this data
into a single number that is easier to grasp. However, the
downside of this approach is that when the data is combined or
composited into the descriptive statistic, information
embedded in the raw data is lost. In the case of any index,
including the API or any of the EQI-related indices, changes
in any one of the dimensions measured in that index will lead
to changes in the value of the index, even if there are no
changes in any of the other dimensions. Effectively this
means that as a greater number of dimensions are composited
into the index, it becomes more difficult to gain a detailed
SB 547
Page 9
understanding of what is really happening at the school or
district from an examination of the value of the index alone.
The index basically becomes a pure number, and the reasons for
that number either rising or falling could be tied to so many
factors (and the interactions of those factors) that drawing
policy conclusions about what works or doesn't in that school
or district may be difficult. This would be especially true
with the EQI, where so many different dimensions, each with a
number of specific pieces of data included, are composited
into the index; the production of the sub-indices (SAI, GRI,
CPI, and CRI) proposed by this bill may mediate this problem
to some effect, but likely not entirely.
3)The bill specifies what dimensions and, in some cases, data
should be included in the various indices in the EQI system,
but does not specify how those elements should be composited
into the indices. Instead the bill leaves much of the
determination of how specified data elements are incorporated
into the various indices (i.e., the methodology for
calculating each index) up to the technical advisory group
that was formed to advise the SPI and SBE on the API. On one
hand, this work is so technical that it should be ministerial
and not incorporated into statute. At the same time, clearly
the results of any accountability index, and thus the
outcomes, rewards, interventions and sanctions that may be
part of the associated accountability system, will depend on
the specific methodology used to make the necessary
calculations. The Legislature may want to consider whether it
wishes to grant complete authority to the SPI and SBE to
review and adopt the methodology for calculating the indices,
and thus for determining how schools and districts fare in any
resulting rankings or outcomes.
4)Though inclusion of graduation rates in the API is in current
law, authority was provided to the SPI to determine when data
on graduation rates would be included in the API; at this
point in time, test scores remain as the only data on which
the API is based. Opponents of past proposals to include
additional data, such as graduation rates or course completion
rates, in the API have argued that data on these other
dimensions suffer from three problems: a) non-testing data are
less objective in nature than testing data, and thus do not
reflect pure academic outcomes; b) data on other dimensions
are not robust enough for this type of use and thus may
reflect very different outcomes in different schools or
SB 547
Page 10
districts; and c) it is unclear how the state would measure or
collect data on certain dimensions, such as whether courses of
study fulfill university admission requirements or how well
schools and districts prepare students for the workplace.
5)With respect to the collection, or the existence of, specific
data elements, which are required by this bill to be included
in the EQI system, it is unclear that all of these data
elements are collected or available, or if they will be
available at any time in the near future. To the extent that
some of these data are not available, that would either reduce
the base of multiple measures included in the EQI or have
clear cost implications for the implementation of this
proposal.
6)Implementation of the EQI system, and the repeal of the
current API, would have ripple effects through the Education
Code (EC) and the programs that are authorized and mandated
therein. A quick search of the EC shows that there are nearly
80 sections of code that reference either the API or the
Academic Performance Index; these references range from
non-substantive mentions to situations where the API is used
to define program eligibility or to condition funding or
compliance with a programmatic requirement. Clearly an
extensive clean-up effort would be needed to conform the EC to
these changes. The implementation timeline in the bill would
require that this clean-up effort occur during this
legislative session.
Committee Amendments: Other issues in the bill can be addressed
through Committee amendments; Committee staff recommends the
following amendments to the bill:
1)This bill proposes the transition between the API and the new
EQI measures to occur between the 2012-13 and 2013-14 fiscal
years. Many aspects of assessment and accountability at the
state and federal level will be in transition in the near
future. It is likely that the reauthorization of the federal
Elementary and Secondary Education Act (ESEA, though the
current authorization is referred to as NCLB) will move
forward during 2013, the main part of the state's testing
program becomes inoperable on July 1, 2013 (though pending
legislation approved by this Committee moves that date to
2014), and a national assessment consortium in which
California is participating will bring its work toward an end
SB 547
Page 11
in 2014. Changes, especially in the underlying assessment
system, should pre-date changes in the state's accountability
system so that the technical advisory committee developing the
methodology for calculating the EQI and its component indices
have knowledge of what the underlying data will look like. In
addition, developing the proposed SAI in a post- ESEA
environment might increase the possibility of developing one
index that satisfies both state and federal needs, thus
eliminating the two measure federal/state approach that has
been confusing in recent years. For these reasons, the bill
should be amended to move the transition timeline such that
the current API becomes inoperable on July 1, 2014, and the
new system comes on line for the 2014-15 fiscal year.
2)If the amendment in 1) above is approved by the Committee,
then conforming amendments to timelines and deadlines
specified in other sections of the bill (e.g., reporting
timelines) should also be made.
3)The bill requires the SPI to provide an annual status report
on various aspects of the EQI system and the related work of
the PSAA advisory committee to the Governor and the
legislature. In order to ensure a full accounting of the
status of the system, technical clarifications concerning
specific report elements should be made.
4)The bill maintains the structure and charge of the PSAA
advisory committee; conforming amendments are necessary to
extend the authority of this committee to include work on the
EQI system that is referenced elsewhere in this bill.
5)By making the current API system inoperable and repealed, the
bill eliminates the Alternative Schools Accountability Model
(ASAM). Under the current system, the API scores of community
schools, continuation high schools and non-public schools that
serve special education pupils are not considered reliable due
to both small numbers of scores and the fact that most pupils
are placed in the schools for less than a year. Accordingly,
the SPI is directed in current law to develop an alternative
accountability system under which these schools may receive an
API score, but are not included in API rankings. The ASAM is
the alternative system developed by the SPI for this purpose.
This bill should be amended to provide an alternative method
for determining the success of alternative schools.
SB 547
Page 12
6)The bill makes a substantive change to reporting requirements
for subgroups that is not consistent with amendments that this
Committee took in support of additional reporting of subgroup
data during the Committee's consideration of SB 512 (Price);
this bill should be amended to maintain the current definition
of a numerically significant subgroup, and thus leave any
statutory changes in this area as an issue for SB 512 to
address.
7)At the request of and as a courtesy to the author, Committee
staff recommends that the bill be amended to make technical
changes to the uncodified findings specified in Section 1 of
the bill.
Previous and related legislation: AB 224, (Bonilla), pending in
the Senate Education Committee, requires the SPI, in
consultation with the SBE, to incorporate both previously
specified, as well as additional, measures of performance into
the API, using the best available data and commencing in the
2012-13 fiscal year. AB 400 (Nunez), vetoed in 2007, was
substantially similar to AB 224. AB 519 (Mendoza), held in the
Senate Appropriations Committee in 2007 but later amended to a
different subject, would have required the incorporation into
the API of data regarding the availability in high schools of a
course of study that fulfills University of California and
California State University admission requirements, and the
submission of a plan for incorporating dropout data into the
API. AB 2167 (Arambula), Chapter 743, Statutes of 2006,
establishes a specific methodology for including graduation
rates, as previously required, in the API; also requires the SPI
to report annually to the Legislature on graduation and dropout
rates in the state. SB 1284 (Scott), held in the Assembly
Appropriations Committee in 2006, would have updated and made
technical amendments to statutes that establish the API. SB 257
(Alpert), Chapter 782, Statutes of 2003, requires the advisory
committee established to advise the SPI on the API to make
recommendations to the SPI on a methodology for generating a
"gain" score measurement to provide more accurate measure of a
school's growth over time. AB 1295 (Thomson), Chapter 887,
Statutes of 2001, makes changes to the API to allow small school
districts to receive an API score, receive growth targets, and
performance awards. SB 1 X1 (Alpert), Chapter 3, Statutes of
1999-2000 First Extraordinary Session, known as the Public
Schools Accountability Act (PSAA), authorizes the state's
current accountability program, including establishment of the
SB 547
Page 13
PSAA Advisory Committee and development of the API.
REGISTERED SUPPORT / OPPOSITION :
Support
America's Edge California
Association of California School Administrators (if amended)
California Agricultural Teachers' Association
California Alliance for the Arts
California Association of School Counselors
California Business Education Association
California Manufacturers and Technology Association
California School Boards Association
Hispanas Organized for Political Equality
Los Angeles Unified School District
Superintendent of Public Instruction Tome Torlakson (Sponsor)
The University of California
Opposition
None on file
Analysis Prepared by : Gerald Shelton / ED. / (916) 319-2087