Study Development and Review Process

Texas fared better than many other states during the recent recession. Nevertheless, when the Texas Legislature convenes in January 2011, its members will face some difficult budget decisions.

Public education spending consumes a large and expanding share of state and local revenues. Its growth has consistently outpacing enrollment growth and inflation. Some school districts, however, manage to achieve strong student performance while keeping spending growth to a minimum.

School districts that operate efficiently – achieving strong academic performance while keeping costs low – offer valuable examples for other districts. Widespread adoption of their strategies could help our state and local governments slow the rapid growth of educational spending while ensuring that Texas high school graduates are ready to succeed in college or the work force.

Project Overview

The 2009 Legislature’s House Bill 3 directed the Comptroller to “identify school districts and campuses that use resource allocation practices that contribute to high academic achievement and cost-effective operations.”¹ In response, the Comptroller’s office created the Financial Allocation Study for Texas (FAST) to examine district and campus resource allocation – and the relationship between these allocations and student achievement.

This proved to be a complex task, as many forces influence student learning, including factors both in and outside school. Similarly, the cost of education is influenced by many factors, some beyond the districts’ control.

Expert Consultants

The research team began by assessing the data sources needed to perform the study required by the 2009 Legislature’s House Bill 3 (H.B. 3). This assessment involved collaboration between the Comptroller’s office and recognized experts in the field, including researchers at some of the state’s top institutions of higher education.

• The University of Texas at Dallas (UTD) Texas Schools Project provided detailed student data compliant with the federal Family Educational Rights and Privacy Act, allowing for analyses of student performance that cannot be made with publicly available data from the Texas Education Agency (TEA).
• UTD’s Dan O’Brien Ph.D., Jim Parsons and Kurt Beron, Ph.D., worked with Comptroller staff to develop new academic outcome measures based on scores from the Texas Assessment of Knowledge and Skills (TAKS) exams. These new indicators measure student academic growth from year to year, allowing for more accurate assessments of student progress.
• Lori Taylor, Ph.D., of Texas A&M University provided expertise on school district costs and produced groupings of fiscal peers for comparative purposes.
• Harrison Keller, Ph.D., of the University of Texas at Austin provided guidance on educational policy and assisted in the development of the study’s methods.

Texas Education Leaders

A Superintendent Advisory Committee representing school district leaders from across the state provided valuable input and practical suggestions for this study. Superintendents and their staffs formed working groups to discuss a series of topic areas important to this study, including:

• school cost drivers, including those outside district control;
• useful and reliable indicators of student performance;
• ways in which districts and campuses can be grouped for comparison; and
• the identification of best practices in school operations.

The Comptroller’s Superintendant Advisory Committee included:

Superintendant	District
David Anthony, Ed.D.	Cypress-Fairbanks ISD
Frank Belcher (now retired)	Canadian ISD
Keith Bryant	Bullard ISD
Gene Buinger, Ed.D.	Hurst-Euless-Bedford ISD
Jesus Chavez, Ed.D.	Round Rock ISD
Mike Feinberg	KIPP Houston (Charter)
Cynthia Garcia, Ed.D.	Driscoll ISD
Lorenzo Garcia, Ed.D.	El Paso ISD
Karen Garza, Ph.D.	Lubbock ISD
Roland Hernandez, Ph.D.	Waco ISD (now with Corpus Christi ISD)
Michael Hinojosa, Ed.D.	Dallas ISD
Daniel King, Ph.D.	Pharr-San Juan-Alamo ISD
Duncan Klussmann, Ed.D.	Spring Branch ISD
Richard Middleton, Ph.D.	North East ISD
Sylvester Perez, Ed.D. (now retired)	Midland ISD
Carrol Thomas, Ed.D.	Beaumont ISD

The Comptroller also met with a school board advisory group of Texas school trustees to discuss the study methodology and provide direction:

Trustee	District
Jim de Garavilla	Silsbee ISD
Karen Ellis	Richardson ISD
Carol Fletcher, Ph.D.	Pflugerville ISD
Israel Hinojosa	Jim Hogg County ISD
Mark Miller	Sealy ISD
Lynn Ramsey	Shamrock ISD
Sarah Winkler	Alief ISD
Cindy Warner	Coppell ISD

The research team met with teachers, principals and other education groups to discuss and address their concerns regarding this project. Those who contributed include:

• Association of Texas Professional Educators
• Bill and Melinda Gates Foundation
• Michael & Susan Dell Foundation
• Regional Education Service Center diectors
• Texas Association of School Administrators
• Texas Association of School Boards
• Texas Association of School Business Officials
• Texas Association of Secondary School Principals
• Texas Charter School Association
• Texas Classroom Teachers Association
• Texas Education Agency
• Texas Elementary Principals and Supervisors Association
• Texas Federation of Teachers
• Texas Institute for Education Reform
• Texas High Schools Project
• Texas State Teachers Association

Independent Review

The FAST project used two types of teams, technical teams and peer-review panels, to validate its methods and findings.

The technical teams provided guidance on the development of academic and financial performance indicators. These teams primarily comprised Texas academic and financial experts, including:

Academic Measures Team

• Chrys Dougherty, Ph.D., National Center for Educational Achievement
• Jon Lorence, Ph.D., University of Houston
• Jim Van Overschelde, Ph.D., Texas Education Agency (now with E3 Alliance)
• Lori Taylor, Ph.D., Texas A&M University
• Dash Weerasinghe, Ph.D., Plano ISD
• Victor Willson, Ph.D., Texas A&M University
• Gloria Zyskowski, Ph.D., Texas Education Agency

Financial Measures Team

• Tom Canby, Texas Association of School Business Officials
• Jim Dyer, Ph.D., McCombs School of Business, University of Texas at Austin
• Timothy Gronberg, Ph.D., Texas A&M University
• Kathy Hayes, Ph.D., Southern Methodist University
• Jim Parsons, Texas Schools Project, UT-Dallas
• R. Anthony Rolle, Ph.D., Texas A&M University (now with University of South Florida)

After the study methods were developed, they were submitted for analysis to a pair of independent peer review panels, one for academic progress and the other for financial and efficiency measures. These panels provided recommendations and comments on the draft methodologies. Their members included:

Peer Review Panel on Academic Measures

• Joan Herman, Ed.D., University of California, Los Angeles
• Michael Podgursky, Ph.D., University of Missouri
• Steven Rivkin, Ph.D., Amherst College
• William Sanders, Ph.D., SAS Institute

Peer Review Panel on Financial Measures

• William Duncombe, Ph.D., Syracuse University
• Stephen Frank, Ph.D., Education Resource Strategies
• Shawna Grosskopf, Ph.D., Oregon State University
• Jennifer Imazeki, Ph.D., San Diego State University
• Andrew Reschovsky, Ph.D., University of Wisconsin
• Amy Schwartz, Ph.D., New York University

Methodology

The FAST team produced methods to place Texas campuses and districts on a level playing field for comparisons of academic performance and spending. Any such measure must weigh certain factors, such as geography and demographics, which are beyond schools districts’ control. The measures of academic growth used in this study control for several of these factors.

The FAST report and an online web tool also provide multiple “lenses” through which to examine campus and district performance. Districts and campuses with similar characteristics can be grouped together for comparison across dozens of academic and financial performance indicators.

The FAST methodology also adjusts for a number of demographic, economic, geographic and other characteristics that affect academic performance and spending. The FAST methodology for measuring academic performance includes 32 control variables while the spending methodology adjusts for eight.

Academic Measures

• The FAST analysis uses a value-added model that measures achievement by controlling for the varying characteristics of students, campuses and districts to estimate how much a district or campus contributes to student learning.
• Using the value-added model, the FAST report measures annual progress in reading/English Language Arts and math.
• The research team developed a composite academic progress rating by combining measures of math and reading progress.
• All academic progress measures are shown in percentiles ranging from one to 99, with 99 representing the most academic progress relative to other districts in the state.
• By controlling for factors outside of teaching that influence student performance, the research team is able to compare academic progress among districts and campuses on a level playing field.
• To further ensure fair comparisons, academic progress is averaged over three years.
• Thus, if a district has a math progress score of 60, it means that during the last three school years, the district’s students showed as much or more progress on math TAKS than 60 percent of districts statewide.

Spending Index

• Texas districts and campuses operate in a variety of “cost environments” – socioeconomic and geographic characteristics that influence the cost of education and are often beyond a school district’s control.
• The research team evaluated financial data for each district and campus by comparing them to “fiscal peers” – districts and campuses that operate in similar cost environments, are of similar size and serve similar students.
• To ensure the validity of financial comparisons, the research team employed a technique called propensity-score matching to identify up to 40 peers for each Texas school district and campus, based on common cost factors such as wages, school district size and geography and student demographics.
• After a group of fiscal peers is identified for a school district, the district is then assigned a “spending index” based on its spending relative to its fiscal peers.
  • In creating the spending index, FAST compares district core operating expenditures per pupil, adjusted for geographic wage variations.
  • A district’s spending index is determined by identifying the spending quintile in which it falls relative to its fiscal peers. The quintiles range from very low to very high, with very low indicating the lowest relative spending in the fiscal peer group and very high representing the highest.
• A similar process is used to create a spending index for each campus.
• There are, however, no uniform standards for districts to follow when allocating expenses to their campuses.
  • Some districts allocate most of their central administration activities to specific campuses, while others do not.
  • Operating expenditures for campus-related activities (instruction, instructional services, school leadership and student support services) are more consistently defined across campuses.
  • Because of this, the FAST spending index for campuses is based only on campus-related expenditures.
• To alleviate the possibility of one-year anomalies in spending having undue influence, the research team used three-year averages for spending comparisons.

FAST Rating

Go Behind the Numbers

A detailed look at the methodology and calculations used to produce the FAST ratings is available in the FAST Study Appendix.(PDF, 4.7MB)

• Finally, the review team created a FAST rating that integrates the academic progress and spending measures to identify districts responsible for strong and cost-effective academic growth.
• Each district has received a FAST rating ranging from one to five stars, with half-star increments (Exhibit 1).
• A five-star district has a composite progress rating between 80 and 99 and a spending index of “Very Low.”
• A one-star district has a composite progress rating below 20 and a spending index of “Very High.”
• A district with “Very High” spending and a composite progress rating of 80 to 99, and a district with “Very Low” spending and a composite progress percentile below 20, both earn three-star FAST ratings.
• This rating does not make any judgment of the relative value of spending versus academic progress, recognizing that different school districts have different priorities and different constraints.