Slide1 : Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning
Norman L. Webb
Robert H. Meyer
Paula A. White
Adding Value to the Mathematics and Science
Partnership Evaluations
Wisconsin Center for Education Research
University of Wisconsin
A Seminar at the University of California-Irvine
October 3, 2003
Slide4 : National Longitudinal Study of Mathematical Abilities (NLSMA)
1962-1967
N=100,000 Students
Grade Levels
4 5 6 7 8 9 10 11 12
X Pop X X X X X
Y Pop X X X X X
Z Pop X X X
General Components of an Education System : General Components of an Education System Management Program Policy Student Outcomes
System Attributes : System Attributes Enabling
Target
Explanatory
Attributes : Attributes Enabling
Capacity
Sustainability
Target
Alignment
Saturation
Quality
Equity
Linkages
Explanatory
Trade-offs
Incentives
Coordinated Linked Capacities Lead to System Fitness for Change : Coordinated Linked Capacities Lead to System Fitness for Change
Slide9 : Components and Elements of Systemic Reform
Slide10 : Components and Elements of Systemic Reform (Continued)
Operationalizing Elements of Systemic Reform (Continued) : Operationalizing Elements of Systemic Reform (Continued) Professional Development
Slide13 : Pre - SSI SSI Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.
Slide14 : Pre - SSI SSI Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.
Slide15 : Pre - SSI SSI Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.
Focus of State SSI and Statewide Achievement Gains from 1992 to 2000 : Focus of State SSI and Statewide Achievement Gains from 1992 to 2000
Slide17 : Basic Evaluation Model for Judging the Impact of a MSP
ΔAchievement = f (MSP) + ε
Slide18 : SCALE Evaluation Indicator System Context Project
SCALE District School Input Teacher Student
Disposition
To Learn Opportunity
To Learn Student
Achievement Capacity Action Student
Outcomes Student
Participation University
Number of Hispanic Students Tested by Cohort and Year : Number of Hispanic Students Tested by Cohort and Year
Slide20 : Number of White Students Tested by Cohort and Year
Slide22 : Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model
Slide23 : From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Black Students Rasch Scale (EV1 Estimates)
Slide24 : From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Hispanic Students Rasch Scale (EV1 Estimates)
Slide25 : From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between Advantaged and Disadvantaged Students (EV1 Estimates)
Slide26 : Evaluation Design
Indicator System
Experimental Design
Quasi-experimental Design
Qualitative Analysis
Focus Study
Slide27 : Experimental Design
Control Group
Random Assignment
Post Measure
Sufficient N for Power Requirements
Quasi-Experimental Design : Quasi-Experimental Design Comparison Group
Pre-Post Measures
Replications
Independent Variables : Independent Variables Teacher Knowledge
Teacher Practice
Curriculum
Student Activities
Support
Resources
Professional Development
Slide30 : SCALE Evaluation Indicator System Context Project
SCALE District School Input Teacher Student
Disposition
To Learn Opportunity
To Learn Student
Achievement Capacity Action Student
Outcomes Student
Participation University