Center on Research and Evaluation (CORE) EMBARGOED UNTIL 12 PM ET WEDNESDAY FEB 20. 2019 TEXAS IMPACT: Evaluation Report For Teach For America February 2019 Annie Wright, Ph.D. Dylan Farmer Yusuf Kara, Ph.D. Yetunde Zannou, Ph.D. Anne Ware, Ph.D. EXECUTIVE SUMMARY The Center on Research & Evaluation (CORE) at Southern Methodist University conducted an impact evaluation of Teach For America (TFA) corps members and alumni teachers in five Texas regions - Austin, Dallas-Fort Worth (DFW), Houston, the Rio Grande Valley (RGV), and San Antonio. This evaluation compares academic outcomes for students of TFA corps members and alumni compared to matched teachers with no TFA affiliation and with commensurate classroom experience. The evaluation replicates some existing TFA evaluations and expands both the scope and the rigor of existing evidence. This evaluation spans six academic years (2011-12 to 2016-17), ten grade levels (3rd grade to 12th grade), and nine content areas (Reading, Math, Science, Social Studies, Algebra 1, English 1, English 2, Biology and U.S. History). It utilizes the State of Texas Assessments of Academic Readiness (STAAR) standardized assessment for the first time in a TFA study. It is also the first evaluation of TFA impact in Texas high schools. Overall, across all content areas, students of TFA-affiliated teachers were as likely as or more likely to pass the STAAR assessment than students of non-TFA-affiliated teachers. These results vary across regions, student demographic subgroups, and grade levels. Evaluation Methodology Impact evaluations demonstrate that a given program works, or that it does not. Additionally, it is critically important to identify how and why programs are working. Specifically, understanding the different conditions under which any given program is effective is critical for continuous improvement for scaling and sustainability of effective programs. This is the approach that CORE utilized in this impact evaluation. This impact evaluation spans six academic years, ten grade levels and nine content areas. Results show that Texas students with a TFAaffiliated teacher are as likely as or more likely to pass STAAR than students with a non-TFAaffiliated teacher in the year that they have that teacher. These results stem from a carefully matched sample and logistic regressions that account for variation in students, teachers and regions when establishing likelihood of passing the state assessment. Analyses results show variation in these academic impacts with some content areas and regions having more TFA advantage than others. Overall, TFA alumni are the most effective group of teachers compared to corps members, underscoring a need to retain TFA alumni as classroom teachers over time. Impact analyses for this study were designed to allow for rigorous comparison of TFA and non-TFA conditions as well as to explore more deeply the various conditions that contribute to relative effectiveness. Similar to previous studies, this evaluation considers the role of factors related to the district, school, teacher and student in explaining variability. Unlike other studies, the data set used for analysis includes matched student and teacher-level data that allows for a more rigorous comparison group match and richer analysis of factors that contribute to differences in TFA effects on student outcomes. The central evaluation question was whether there was a differential impact on academic outcomes attributable to having a TFA corps member or alumna as the classroom teacher of record for one academic year. CORE carefully described the variables that contribute to this impact such that deeper knowledge about the mechanisms of TFA impacts in different regions, and for different content areas, grade levels, school type and student demographics is illuminated. CORE also explored the potential influence of “saturation” of TFA in a given context, hypothesizing that there could be setting-level effects of clustering TFA corps members and/or alumni that indirectly influence student outcomes. Two aims for the methodology were paramount: (1) ensuring comparison group equivalence such that the effects of TFA could be appropriately isolated and (2) succinctly and accurately communicating findings from a large number of underlying analyses. Equivalence across the comparison groups began with a strategic data selection process where rules for inclusion were carefully chosen and documented. This initial process removed outliers and ensured that the sample for analyses included teachers of record who were in the classroom for a full year, among other key considerations. Additionally, a rigorous propensity score weighting process helped support confidence in claims of impact by controlling for covarying conditions including characteristics of students, teachers, schools and districts. Logistic regressions then assigned likelihood of passing STAAR to the TFA or non-TFA conditions. In tandem, these analytic strategies allowed for isolation of the effects of having a TFA corps member or alumna in the classroom compared to a non-TFA-affiliated teacher. Due to the ii large number of underlying analyses associated with this expansive dataset, CORE adopted a meta-analysis strategy and averaged the likelihood of passing STAAR in a given content area with a TFA-affiliated teacher (either corps member or alumni) or a non-TFA-affiliated teacher. Summary of Key Findings A synthesis of 699 underlying analyses that combined grade levels, content areas, and academic years showed that the average probability of passing STAAR within the academic year that a student was with a TFA-affiliated teacher was 39.3% compared to a matched sample of students with non-TFA-affiliated teachers which was a 36.8% probability. There is variability within these overall trends, and each are summarized below and in the full report. However, based on the metaanalysis, the takeaway is that students of TFA-affiliated teachers are overall, slightly more likely to pass STAAR in the academic year in which they had the TFA corps member or alumni teacher. Student Demographics and School Type: • The advantage of having a TFA-affiliated teacher is the same for economically disadvantaged and non-economically disadvantaged students. • There is an advantage of TFA for students of all races; the effect is strongest specifically for Black and Hispanic students who have a TFA alumni as their teacher. • There is an advantage of TFA in both traditional public ISDs and charters; the effect is strongest in ISDs. • There is an advantage of TFA in schools that meet and do not meet accountability standards; the effect is strongest in schools with low state accountability ratings. There is an advantage of TFA for both Limited English Proficient (LEP) and non-LEP students; the effect is strongest for LEP students. Relative benefits for having a TFA corps member or alumni teacher were seen for key groups of Texas students who best represent TFA’s overall mission of education equity for all. That is, TFA-affiliated teachers are more effective than non-TFA-affiliated teachers, on average, for all subpopulations. However, this relative advantage was greater and more consistent for students in traditional public schools than in charter schools, for Black and Hispanic students than for White students, for LEP students than non-LEP students, and for students at schools not meeting state performance standards than at schools that are meeting standards. TFA-affiliated teachers are equally effective as non-TFA-affiliated teachers, on average, for both economically disadvantaged and non-economically disadvantaged students. Teacher Experience: • Findings differed by TFA affiliation (new corps members compared to corps members compared to alumni); the advantage of TFA is strongest with TFA alumni. Compared to brand new TFA corps members and all corps members, TFA alumni offer the best advantage to students in terms of academic outcomes, underscoring the importance of retaining alumni as classroom teachers. Students of TFA alumni are 7.2% more likely, on average, to meet state standards than students of comparable veteran non-TFA-affiliated teachers. This likelihood is followed by brand new corps members in their first year (0.3% advantage). There is relatively no advantage of TFA for all corps members (0.1% difference), indicating that a student of a TFA corps member is as likely to meet state standards as a student of a novice non-TFA-affiliated teacher. This same trend is true across all Texas geographic regions included in the study, all content areas, all student groups (including race/ethnicity, LEP and economic disadvantage), and school types (charter/traditional, met standard/did not meet). Content Area: • There is an advantage of having a TFA-affiliated teacher across all 9 content areas. • The largest advantage of TFA was in high school subjects. • In elementary and middle school, the relative advantage of TFA is greater for Math than Reading (TFA and nonTFA-affiliated teachers were nearly equally effective for Reading). When examining content areas, it is clear that students of TFA-affiliated teachers receive the greatest benefit in U.S. History, followed by Biology, English 2, Algebra 1, Math (grades 3-8), Science (grades 5 & 8) and finally, Reading (grades 3-8). The content area with the least overall advantage is Reading, with students of TFA-affiliated teachers being slightly iii more likely to pass the STAAR Reading state assessment. However, the difference is slight with students of TFA-affiliated teachers 0.9% more likely to pass Reading. The greatest single benefit across any region, content area, or grade level is in the RGV, where 8th grade students of TFA alumni are 33.2% more likely to pass STAAR Reading than students with a comparable (veteran) non-TFA-affiliated teacher. This evaluation is one of the first to examine TFA impacts on high school student achievement and results are positive. The advantage of having a TFA corps member or alum teacher in the high school core subjects surpasses the advantages in elementary and middle school grades. Overall, having a TFA-affiliated teacher in U.S. History has the greatest advantage, followed by Biology, English 2 and Algebra 1. Implications This evaluation provides a rigorous look at TFA impacts over multiple years and across a broad geographic area, the largest of all TFA regions. It has strong implications for public and private agencies seeking to continue or expand support for the TFA model and contains important feedback for TFA as they continue to support implementation of the model in a diverse array of schools, districts and communities. One particularly salient outcome is the demonstrated effectiveness of TFA alumni compared to corps members in their first or second year; efforts to keep TFA alumni in classrooms are warranted. Strong evaluation is an ongoing process and no one report will definitively answer all available questions. In this initial impact report, CORE and TFA have laid the groundwork for a number of new evaluation questions. The dataset from Texas Education Agency (TEA) combined with internal TFA documentation will allow for a number of ongoing exploratory analyses that can inform immediate problems of practice as well as future decision-making for TFA and other educational stakeholders. For instance, CORE plans a series of longitudinal questions that will take advantage of the multiple years of available data to ask and answer questions about (1) contributing influence of students’ prior academic performance, (2) potential sustained effects over time of having a TFA corps member or alumni teacher and (3) effects of having a TFA-affiliated teacher for multiple years in a row. These analyses will require identifying “good track” data – student level records that can be followed over multiple years, linking students to key outcome and covariate conditions. CORE will continue to explore the initial findings that indicate an association between TFA campus-level saturation and student outcomes. In addition, CORE and TFA both have a keen interest in exploring the existing data in order to account for additional individual and setting-level variation in students’ and teachers’ experiences. iv Table of Contents EXECUTIVE SUMMARY ............................................................................................................................................................................ ii Introduction ....................................................................................................................................................................................................... 1 About CORE................................................................................................................................................................................................. 1 About Teach For America .......................................................................................................................................................................... 1 About the Evaluation ................................................................................................................................................................................... 1 Background ........................................................................................................................................................................................................ 3 Prior Studies................................................................................................................................................................................................... 3 The Current Study ........................................................................................................................................................................................ 5 Methods .............................................................................................................................................................................................................. 6 Data Management......................................................................................................................................................................................... 6 Data Selection and Request .................................................................................................................................................................... 6 Data Cleaning/Preparing for Analysis .................................................................................................................................................. 7 Sample for Analysis ...................................................................................................................................................................................... 7 Analysis Procedure ....................................................................................................................................................................................... 9 Logistic Regression................................................................................................................................................................................... 9 Propensity Score Weighting .................................................................................................................................................................. 10 Saturation ................................................................................................................................................................................................. 11 Results ............................................................................................................................................................................................................... 12 Impact of TFA ............................................................................................................................................................................................ 12 Layout of Results Section ...................................................................................................................................................................... 12 Synthesis of All Analyses ....................................................................................................................................................................... 13 Reading ..................................................................................................................................................................................................... 17 Math .......................................................................................................................................................................................................... 18 Science ...................................................................................................................................................................................................... 20 Social Studies ........................................................................................................................................................................................... 21 High School End of Course (EOC) Exams ....................................................................................................................................... 23 Discussion ........................................................................................................................................................................................................ 28 Limitations of the Study ............................................................................................................................................................................ 28 Future Research .......................................................................................................................................................................................... 29 References ........................................................................................................................................................................................................ 30 Appendix A: Details of Data Selection .......................................................................................................................................................... I Appendix B: Exploration of Effects of Campus Level TFA Saturation on Student Achievement ................................................... II Appendix C: Results of ANOVA Meta-Analysis of all 699 Impact Analyses ......................................................................................VI Appendix D: Comparison Analysis Heat Map......................................................................................................................................... VII Appendix E: Summary of Comparison Analysis Findings by Demographic Subgroups................................................................ VIII v Introduction About CORE A nationally ranked private university with seven degree-granting schools, Southern Methodist University (SMU) is a distinguished center for teaching and research. Housed within SMU’s Simmons School of Education and Human Development, the Center on Research and Evaluation (CORE) provides a range of research, evaluation, and consultation services. CORE’s overall aim is to use evaluation science to improve educational outcomes for youth. Many of CORE’s projects center on early childhood, out-of-school learning opportunities, and educator preparation and professional development initiatives. CORE’s work emphasizes the community contexts in which these educational initiatives are implemented as a key consideration for understanding effectiveness. CORE staff are interdisciplinary, representing educational research and evaluation, quantitative methodology, psychology and social work and many of CORE’s evaluators have classroom teaching experience. About Teach For America Teach For America (TFA) is a national organization that recruits leaders early in their careers to teach for two years in one of 51 urban and rural regions across the U.S. According to the TFA Foundations document, 1 their mission is to find, develop, and support a diverse network of leaders who expand opportunity for children from classrooms, schools, and every sector and field that shapes the broader systems in which schools operate. In partnership with schools, families, local universities, other organizations, and businesses in the community, TFA provides initial training, ongoing professional development, and access to a resource and support network for corps members and alumni. Some TFA alumni choose to teach in high-needs schools and communities beyond their two-year commitments. Others lead from many sectors that influence the context and conditions of schools in education leadership, policy, advocacy, social entrepreneurship, and as business, philanthropic, and civic leaders. TFA’s presence is greatest in Texas. In 2013, 1,206 corps members worked in Dallas, Houston, San Antonio, and the Rio Grande Valley compared to 782 in the next largest state of California (Vasquez Heilig, & Jez, 2014). Today, in addition to the four Texas regions above, TFA partners with school districts in the Austin region, together representing eight counties consisting of approximately 20 school districts. About the Evaluation Impact evaluations typically aim to demonstrate that a given program works, or that it does not. Additionally, it is critically important to identify how and why programs are working. Specifically, understanding the conditions under which a given program is effective is critical for continuous improvement CORE’s evaluation examined and scaling of effective programs. CORE’s evaluation of TFA was designed overall impact of TFA on to accomplish both of these aims. academic outcomes within one year. It replicates and expands on existing evidence about TFA impacts and describes the various conditions that differentially contribute to effectiveness in various Texas settings. The impact analyses in this evaluation replicate and expand on existing evidence of TFA impact by allowing for rigorous comparison of TFA and non-TFA conditions. A growing body of empirical evidence suggests that students of TFA corps member or alumni teachers outperform or perform as well as students of non-TFA-affiliated teachers based on subject matter, grade level, and student and teacher characteristics. Evaluation and research studies of the impact of TFA on student outcomes in Texas support these findings, but have limitations related to sample size, content area, student characteristics, and outdated state assessments to measure student achievement (Houston Independent School 1 R. Carreon, personal communication, January 7, 2018 1 District [HISD], n.d., 2018; Mickelson & McEnturff, 2015; Raymond, Fletcher, & Luque, 2001; Turner, Goodman, Adachi, Brite, & Decker, 2012; Ware et al., 2011). This evaluation addresses these limitations; it replicates existing findings and expands both the scope and the rigor of existing evidence. Similar to other studies, this evaluation considers the role of factors related to the region, school, teacher and student in explaining variability. Unlike other studies, the data set used for analysis includes deidentified, matched student and teacher-level data that allows for a more rigorous comparison group match and richer analysis of factors that contribute to differences in student outcomes. This evaluation spans six academic years (2011-12 to 2016-17), ten grade levels (3rd grade to 12th grade), and nine content areas (Reading, Math, Science, Social Studies, Algebra 1, English 1, English 2, Biology and U.S. History). This evaluation extends findings from earlier studies by using the State of Texas Assessments of Academic Readiness (STAAR) standardized assessment as an outcome measure and examining TFA impact in high school, both for the first known time. In addition to establishing robust understanding of impact at scale, this evaluation explores more deeply the various conditions that contribute to effectiveness. This evaluation includes a thorough description of variables that contribute to impacts such that deeper knowledge about the mechanisms of TFA impacts in different regions, and for different content areas, grade levels, school type, and student body composition is illuminated. Additionally, analyses explore the potential influence of “saturation” of TFA in a given context, hypothesizing that there could be setting-level effects of clustering TFA corps members and/or alumni that indirectly effect influence student outcomes. 2 Background Prior Studies General findings. A review of the literature points to strong evidence of TFA impact, where students of TFAaffiliated teachers outperform students of non-TFA-affiliated teachers in relatively well-designed studies. Some studies employed a rigorous random assignment design (Clark, Isenberg, Liu, Makowsky, & Zukiewicz, 2017; Clark et al., 2013; Glazerman, Mayer, & Decker, 2006), but most others employed a quasi-experimental design that created statistically matched groups of students or used statistical analyses to control student and/or teacher characteristics (Hansen, Backes, Brady, & Xu, 2015; Henry et al., 2015; Turner et al., 2012; Ware et al., 2011). For outcome measures, these studies focused primarily on student achievement in Mathematics and Reading/English Language Arts, while only a few included Social Studies (Henry et al., 2014) and/or Science (Penner, 2016; Xu, Hannaway, & Taylor, 2007). Studies included a wide range of grade levels and spanned one academic year through as many as six years. Because TFA-affiliated teachers commit to two years of classroom teaching, most studies compared the academic achievement of students taught by TFA corps members with those of students taught by novice non-TFA-affiliated teachers. Some of the studies also examined student outcomes for students of TFA alumni and experienced non-TFA-affiliated teachers (Hansen et al., 2015; Turner et al., 2012). In a few studies, researchers also examined student outcomes in relation to student characteristics, such as race/ethnicity and prior academic achievement (Decker, 2004; Ware et al., 2011). Overall, the results of these prior studies suggest that TFA Mathematics teachers are more effective than their nonTFA counterparts, but that TFA and non-TFA-affiliated Reading/English Language Arts teachers perform about the same. In a rigorous review of seven studies investigating the effects of TFA-affiliated teachers on student outcomes, the What Works Clearinghouse (WWC) concluded that TFA-affiliated teachers have “positive effects on Mathematics achievement, potentially positive effects on Science achievement, and no discernible effects on Social Studies achievement and English Language Arts achievement” (United States Department of Education [USDOE], 2016, p. 1). The conclusions about Science and Social Studies are due, in part, to the lack of focus on these content areas in most of the studies, but the conclusions about Mathematics and Reading/Language Arts are based on strong evidence across all of the studies. Additional studies not included in the WWC review confirm the conclusions with at least five reporting a positive effect of TFA-affiliated teachers on student Math achievement (Decker, Mayer, & Glazerman, 2004; Hansen et al., 2015; Mickelson & McEnturff, 2015), but only Mickelson and McEnturff (2015) identified a positive effect for TFA-affiliated teachers in Reading/Language Arts. Teacher experience. As noted above, the studies included in this review varied in terms of their focus on different student grade levels, and student characteristics as well as the composition of the teacher groups (e.g., novice, first year, second year, experienced) used in the comparisons of TFA and non-TFA-affiliated teachers. As a result, the findings are mixed over and above the general conclusions related to student Math and Reading/Language Arts achievement. To account for the fact that TFA corps member teachers have two or fewer years of experience, all of the studies reviewed for the current report compared the performance of TFA-affiliated teachers with the performance of novice non-TFA-affiliated teachers. Several of these studies also compared the TFA-affiliated teachers with all non-TFA-affiliated teachers in the study (Clark et al., 2017; Decker, et al., 2004) or controlled for teacher experience in their analyses (Hansen, et al., 2015). Considered together, these studies provide some evidence that TFA-affiliated Math teachers outperform their non-TFA counterparts at each experience level. For example, experimental studies found a positive impact for TFA-affiliated Math teachers relative to the novice control teachers and all control teachers (Decker et al., 2004; Glazerman et al., 2006; Clark et al., 2013). Turner et al. (2012) used statistically matched student groups to compare novice TFA-affiliated teachers with novice non-TFA-affiliated teachers and to compare veteran TFA-affiliated teachers with veteran non-TFA-affiliated teachers. They found positive effects for TFA alumni at the middle school level for English Language Arts and for novice TFA-affiliated Math teachers, when compared with teachers of similar experience. Finally, Hansen et al. (2015) and Henry et al. 3 (2014) included teacher experience as covariates in their analyses. Hansen et al. (2015) found similar positive effects for TFA-affiliated Math teachers, while Henry et al. (2014) found that the performance of TFA and non-TFAaffiliated teachers improved similarly as each group gained experience. With these mixed results from prior studies, additional evidence is still needed to determine if there is any clear pattern of effects for TFA and non-TFA-affiliated teachers with different levels of experience. Student Characteristics. In addition to examining the impact of TFA for different types of teachers, most of the studies also considered variations in student characteristics in their analyses. Most studies report findings for students at different grade levels, but few report findings for different student demographic groups. In their experimental study, Decker et al. (2004) found that the positive effects of TFA-affiliated elementary school Math teachers were similar regardless of student ethnicity, grade level, and gender. Similarly, though not an experimental study, Hansen et al. (2015) reported a positive impact of TFA-affiliated teachers when controlling for student gender, ethnicity, free/reduced lunch status, and grade level in a series of value-added regressions. In consideration of at-risk student characteristics, Ware et al. (2011) reported greater Math gains for African American and Hispanic students of high school TFA-affiliated teachers but not for minority students at the middle school level. An evaluation of TFA in Houston ISD found that economically disadvantaged students of TFA-affiliated Math teachers had higher passing rates on the state Math assessment than their counterparts taught by non-TFA-affiliated teachers in each of the three years of the evaluation (HISD, 2018). For Reading, however, the passing rates of the economically disadvantaged students of TFA-affiliated teachers were greater in only one year and similar in the other two years. Considered together the studies suggest that the observed effects of TFA-affiliated teachers are over and above varying student characteristics and point to potential impacts on specific groups of students. More comprehensive studies need to be conducted to better understand these potential effects of TFA-affiliated teachers on different groups of students, particularly those at risk. TFA in Texas. Of particular importance for the current evaluation are the studies that have examined the effects of TFA-affiliated teachers in Texas and the gaps in evidence and understanding that have not yet been addressed. All of the Texas-based studies were quasi-experimental, as is the current study; four were district level evaluations (Raymond et al., 2001; Mickelson & McEnturff, 2015; HISD, n.d., 2018) and two focused on multiple school districts and/or regions in Texas (Turner et al., 2012; Ware, et al., 2011). In general, these studies supported other findings that TFAaffiliated Math teachers outperformed their non-TFA-affiliated colleagues (at varied grade levels and teacher experience levels), and also provided some evidence of a positive impact for Reading/English Language Arts teachers. These Texas studies, however, are somewhat limited in their scope, as all studies necessarily are. These limitations highlight gaps that CORE sought to address in the current design. For example, the district level evaluations focused on only one district and cannot necessarily be generalized to other districts or regions in Texas. Turner et al. (2012), the most comprehensive study of TFA in Texas, includes teacher and student data from 483 campuses in four distinct Texas regions. However, the study only assesses student outcomes for one academic year, and did not assess outcomes for high school students. Ware et al. (2011) included two academic years in their analyses, but focused on only four Texas school districts, again making it difficult to generalize across districts or to identify any potential differences in Texas regions. Neither of these studies examined achievement in Science or Social Studies. Finally, with two exceptions (Mickelson & McEnturff, 2015; HISD 2018), the outcome measures of these Texas studies focused on state accountability assessments that have since been replaced by the newly adopted STAAR. STAAR was intended to be more rigorous, required more critical thinking and, necessarily had different passing standards than the earlier assessments. To date, no statewide study of TFA has been conducted using the new STAAR assessments as outcome measures. The two studies that have assessed STAAR results are district level evaluations. Though not generalizable to other Texas regions, they provide some initial evidence of TFA-affiliated teacher effects for different subject areas, teacher experience levels, and student grade levels. The Dallas ISD evaluation focuses on only one year of STAAR results with regression analyses that reveal higher Math passing rates for 3rd - 5th grade students of TFA-affiliated teachers than non-TFA-affiliated teachers at each level of teacher experience, as well as 4 higher passing rates for Math middle school and Algebra 1 students of 2nd year TFA-affiliated teachers (Mickelson & McEnturff, 2015). Similarly, in its evaluation covering three years of STAAR results, Houston ISD (2018) found that TFA-affiliated Math teachers (grades 3–8) outperformed non-TFA-affiliated Math teachers in each of the three years while the TFA-affiliated Reading teachers had positive outcomes in one year and performed similarly to the nonTFA-affiliated teachers in the other two years (HISD, 2018). The Current Study The current study expands on existing literature and identifies additional patterns across regions, schools, teachers and students. Most significantly, previous studies estimated the effects of TFA by aggregating student achievement outcomes and attributing them to the school where TFA-affiliated teachers worked, in lieu of linking specific students with teachers—TFA and non-TFA. This study overcomes that challenge using student and teacher-level data that are linked, but are deidentified and provided by Texas Education Agency (TEA), thereby maintaining confidentiality and impartiality in the analysis. Additionally, the current study is the first comprehensive study of TFA to use STAAR; the most immediate prior studies used the Texas Assessment of Knowledge and Skills (TAKS), which was phased out and replaced by STAAR from 2012 to 2014 2. This study addresses key limitations in prior studies of the TFA impact on student achievement in Texas by: (1) using STAAR exams as an indicator of student achievement, (2) considering school and district characteristics in estimates of the effect of TFA corps members and alumni teachers on student outcomes, and (3) linking de-identifiable student-level data with de-identified teachers, by grade level and course. 2 The STAAR assessment represents increased rigor compared to TAKS. The content and skills assessed by STAAR require a higher level of complexity and more authentic application of content and skills (e.g., less multiple choice items). Also, the TAKS assesses general knowledge that would accumulate across multiple school years, while STAAR goes into more depth on grade-specific content and skills. Results from analyses indicate that STAAR is more difficult to TAKS and that students are likely to answer fewer questions correctly on STAAR than on TAKS. Additionally, the STAAR assessment has more items and a shorter time limit. https://tea.texas.gov/student.assessment/staar 5 Methods The geographic focus of this evaluation is five identified Texas regions—Austin, Dallas-Fort Worth (DFW), Houston, Rio Grande Valley (RGV), and San Antonio, covering 8 counties—and the regional Independent School Districts (ISDs) and charter school systems in the counties in these regions. The grade levels and subjects tested in STAAR form the key outcomes of interest. These are: Reading and Math (grades 3-8), Science (grades 5 and 8), Social Studies (grade 8), and high school end of course exams for Algebra I, English I and II, Biology and U.S. History. STAAR Writing was excluded from these analyses due to the nature of that specific exam. Data Management Data Selection and Request SMU’s Institutional Review Board approved the study as exempt. All study data is and was deidentified. No individual student, teacher nor school or campus can be identified. CORE received all data for the planned analyses from the TEA via a Public Information Request (PIR) as allowed by The Texas Public Information Act. CORE initially corresponded with staff in TEA’s Division of Research and Analysis, who recommended some key parameters for the project. First, in order to protect the anonymity of teacher data, TEA recommended that while CORE would manage the overall data request, TFA should provide teacher ID numbers directly to TEA, bypassing CORE. Thus, CORE never had in its possession data that will allow an individual teacher to be identified. Second, the analyses for this study did not require CORE to receive identifiable student level data; thus, student-level data was de-identified. Once parameters were in place, CORE issued a formal request to TEA for the needed data. Specifically, CORE requested STAAR test results for all students enrolled in all school districts across eight Texas counties in the aforementioned regions during the 2011-12 through 2016-17 school years. The counties included Travis, Dallas, Tarrant, Harris, Cameron, Hidalgo, Starr, and Bexar. Individual district indicators were not requested. In addition to de-identified student-level STAAR results, each student’s test record also included information about the teacher of record for that course in that school year and the student’s school. Each teacher of record was flagged as either a TFA corps member in the school year of record, a TFA alumni, or a non-TFA-affiliated teacher. All data were organized at the student level, with assessment, teacher and school level indicators matched at the student level. Data about Students For all students, CORE requested an encrypted identification number, school year, grade level, economically disadvantaged status, LEP status, special education status, gender, age, bilingual education status, race, ethnicity, TEA at-risk status, and the encrypted ID for the teacher of record tied to each of the students’ STAAR assessments in a given year. STAAR data included Reading, Mathematics, Science, Social Studies, and Writing (excluded from the analyses) assessments, as well as end of course exams for Algebra 1, Biology, English 1, English 2, and U.S. History. Specific STAAR metrics included language, test version (e.g., accommodations), raw score, scale score, growth index, and proficiency index. Data about Teachers In addition to the encrypted teacher ID for each teacher of record associated with a student STAAR score, CORE also received race, ethnicity, the flag for TFA affiliations, and years of teaching experience. Grade level taught was derived from the student enrollment data. Data about Schools and Districts While CORE did not request data classifying the identity of specific schools or districts, some information about the students’ schools and districts was received. This included whether the student’s campus was a charter or traditional ISD, the TEA school rating, number of TFA and non-TFA-affiliated teachers on the campus, percent of free/reduced lunch students on the campus, percent of students classified by race on the campus, and grade levels served. CORE 6 also requested the district accountability ratings and number of performance indicators met by the district in which each student was enrolled. Data Cleaning/Preparing for Analysis CORE received two types of datasets from TEA. The first dataset was demographic data, including students, teachers, and campus-level indicators. The second data set contained student STAAR achievement data. Datasets covered six academic years (2011-12 to 2016-17), ten grade levels (3rd to 12th), and nine subject areas (Reading, Mathematics, Science, Social Studies, Algebra 1, English 1, English 2, Biology, and U.S. History). Demographic datasets were cleaned and merged to form a single student-level final dataset for the statistical analyses. Thus, all analyses were conducted by using the student-level observations, and all campus and district level data such as school demographics and campus accountability rating were disaggregated to the student level. Finally, students’ STAAR test data were merged to the final demographic dataset by using the encrypted student ID numbers. Inclusion Criteria For impact analyses, the study sample includes: • Teachers that have a unique TFA/non-TFA affiliation indicator within an academic year • TFA corps members that had maximum two years of experience • Only TFA and non-TFA-affiliated teachers of records (excludes teaching assistants and aides) • Districts that have at least one TFA-affiliated teacher in any of the six academic years • Students that did not change their schools within an academic year • Students taught by a single teacher of a specific subject area (excludes co-teaching cases) • Students that only took the standard version of the STAAR test • For multiple-year analyses, students that did not repeat the same grade During the matching of student and teacher data, CORE used the encrypted class, teacher, and student ID numbers. For grades 3-5, student observations that only matched to a single teacher ID for all subject areas were used for the analyses. In other words, the final dataset for grades 3-5 contained the students taught all core subjects (Reading, Math, Writing, and Science) by the same teacher, during an academic year. For grades 6-12, CORE created four “check variables” in order to mark the unique teacher of the students for each of the subject areas (English Language Arts, Mathematics, Science, Social Studies and end of course subjects in high school) within an academic year. Depending on the analysis condition for the outcome, CORE used these check variables to select the student assessment records that were able to be confidently matched to a single teacher of record that taught the tested subject during that school year. Details about data cleaning and exclusions are in Appendix A, along with the descriptive numbers regarding all data cleaning procedures. Sample for Analysis After preparing data for analyses and omitting student records as needed, a total of 7,298,318 student observations (not unique number of students) were used in the analyses. These observations are distributed across ten grade levels, eight Texas counties (representing the five regions under study), and six school years. CORE’s analyses focused on single-year observations of student performance, not longitudinal analyses. Therefore, an individual student’s performance within a school year is considered a single observation in the sample. That same student’s performance the following school year represents a separate observation. As described in detail in the Analysis Procedure section, evaluation questions were answered by conducting hundreds of individual analyses for sub-samples of the entire sample. Table One describes the demographic characteristics of all students included in the analyses by school year. Table Two describes this same sample of students by geographic regions used for the analyses. 7 Table One. Student sample demographics by school year AY2011-12 n % Total Gender AY2012-13 n % AY2013-14 n % AY2014-15 n % AY2015-16 n % AY2016-17 n % 1,192,774 100 1,207,608 100 1,212,302 100 1,214,361 100 1,197,446 100 1,273,827 100 Male 602,345 50.5 608,818 50.4 611,703 50.5 610,403 50.3 601,982 50.3 646,727 50.8 Female 590,429 49.5 598,790 49.6 600,599 49.5 603,958 49.7 595,464 49.7 627,100 49.2 Black 182,319 15.3 181,730 15.0 178,975 14.8 179,898 14.8 176,851 14.8 187,996 14.8 Hispanic 709,146 59.5 728,799 60.4 738,961 61.0 742,746 61.2 735,629 61.4 791,652 62.1 White 235,114 19.7 230,304 19.1 225,007 18.6 219,828 18.1 211,873 17.7 214,794 16.9 Other 66,195 5.5 66,775 5.5 69,359 5.7 71,889 5.9 73,093 6.1 79,385 6.2 156,675 LEP* 767,837 Economically Disadvantaged Grade Level 13.1 156,680 13.0 168,630 13.9 180,663 14.9 183,745 15.3 226,541 17.8 64.4 783,601 64.9 791,842 65.3 777,727 64.0 771,028 64.4 833,538 65.4 Race ES (3rd-5th) 193,543 16.2 185,966 15.4 176,295 14.5 167,749 13.8 148,259 12.4 146,136 11.5 MS (6th-8th) 445,434 37.3 454,688 37.7 464,763 38.3 463,596 38.2 464,713 38.8 503,920 39.6 HS (9th-12th) 553,797 46.4 566,954 46.9 571,244 47.1 583,016 48.0 584,474 48.8 623,771 49.0 *LEP=limited English proficiency status Table Two. Student sample demographics by geographic region Region A n % Region B n % Region C n % Region D n % Region E n % Gender Male 82,594 51.1 405,625 50.8 424,226 51.0 170,729 50.9 193,169 51.2 Female 79,152 48.9 393,392 49.2 407,271 49.0 164,598 49.1 184,185 48.8 Black 18,534 11.3 186,130 22.9 166,547 19.8 815 0.2 29,519 7.7 Hispanic 88,069 53.8 379,538 46.7 438,933 52.1 326,165 96.9 262,487 68.8 White 44,715 27.3 184,639 22.7 170,992 20.3 7,281 2.2 70,583 18.5 Other 12,280 7.5 61,648 7.6 65,427 7.8 2,389 0.7 18,973 5.0 ES (3rd-5th) 53,341 23.2 195,498 17.3 144,725 12.3 124,264 24.7 164,566 29.4 MS (6th-8th) 84,244 36.6 449,092 39.6 494,185 41.9 184,498 36.6 190,214 34.0 HS (9th-12th) 92,778 40.3 488,075 43.1 539,208 45.8 194,817 38.7 204,744 36.6 Race Grade Level Table Three describes the final sample of teachers retained for the analyses. The final sample included 2,477 active TFA corps members (during any year used in the analyses), 1,479 TFA alumni, and 109,649 non-TFA-affiliated comparison teachers. 8 Table Three. Teacher sample by TFA affiliation status Corps Members n % Alumni n Non-TFA % n % Race Black 410 16.6 247 16.7 16,943 15.5 Hispanic 485 19.6 262 17.7 31,208 28.5 White 1,308 52.8 820 55.4 56,332 51.4 Other 274 11.1 150 10.1 5,166 4.7 361 14.4 197 12.6 25,872 22.6 1,157 46.0 740 47.3 44,322 38.7 997 39.6 628 40.1 44,443 38.8 Region A - 0.0 128 8.6 8,291 7.6 Region B 900 36.4 349 23.4 34,767 31.7 Region C 747 30.2 619 41.5 35,793 32.7 Region D 391 15.8 216 14.5 14,736 13.4 Region E 434 17.6 178 11.9 16,031 14.6 Grade Level Elementary (3rd-5th) Middle (6th-8th) High School (9th - 12th) Geographic Region Note: Total count of teachers within each group fluctuate between race, grade level and area due to missing data. Analysis Procedure In order to determine TFA’s impact on student achievement as well as to identify and succinctly communicate contributing conditions, CORE conducted several types of analyses: • • • logistic regressions in order to assign the likelihood of passing STAAR to either a TFA or non-TFA condition propensity score weighting in order to control for covariate effects by matching two groups of students meta-analysis strategy to assign overall likelihood of passing into categories; identifying the percent of all analyses that showed a statistically significant likelihood in favor of TFA • meta-analysis strategy using analysis of variance (ANOVA) to determine what broad factors (e.g., region, content area, or grade level) were associated with differentiated impact of TFA • descriptive and comparative analyses of likelihoods in order to synthesize trends All conditions (i.e., content area, geographic area, school year, and grade level) were considered as varying conditions. However, not all possible combinations of the various factors were used for final analyses due to incidences of low sample size. For example, measuring the effect of TFA on Social Studies achievement for one grade level, within one year, within one geographic area, for only students meeting specific demographic criteria, may have resulted in an ultimately small sample size of students taught by TFA-affiliated teachers. Using the various combinations of factors, the study includes a total of 699 unique comparison analyses (with the exclusion of analyses with low-sample sizes and perfect probability issues). Logistic Regression These analyses were designed to determine the likelihood that a student would meet grade level standards if they have a TFA corps member or alumni. The outcome measure for academic achievement was the dichotomized pass/fail indicator for a specific STAAR tested subject. CORE employed the logistic regression (LR) modeling technique to predict a student’s pass/fail condition by using the dichotomous TFA indicator. The model predicted a student’s passing probability for a specific subject, depending on whether they taught by a TFA-affiliated teacher or not. 9 Propensity Score Weighting In order to control the potential covariate effects, such as being economically disadvantaged, CORE adopted the propensity score weighting (PSW) approach. Different from the propensity score matching (PSM) procedure, PSW uses the predicted propensity scores to calculate sampling weights for each of the data observations (Leite, 2017). The PSW method then derives the so-called “weight” scores for all comparison group students depending on their covariate measures and the treatment indicator—in this study, being taught or not being taught by a TFA-affiliated teacher. Different treatment effect evaluations require different weight score calculations by PSW. CORE preferred the “average treatment effect on the treated (ATT)” (Harder, Stuart, & Anthony, 2010), where all students taught by a TFAaffiliated teacher have a weight of one. All students taught by a non-TFA-affiliated teacher have weights lower than one depending on their measures of the multiple covariates used to estimate the propensity scores. Higher weight scores for those students indicate a better degree of matching with the students taught by a TFA-affiliated teacher in terms of the covariates. Using the ATT warrants inclusion of the entire group of students taught by a TFA-affiliated teacher (since their weights are all one) whose sample size are considerably lower than the students taught by a nonTFA-affiliated teacher. CORE identified different possible analysis conditions by using multiple factors such as, but not limited to, region, academic year, subject of interest, and grade level. All condition-specific analyses were conducted separately for three treatment conditions: (1) brand-new TFA corps members in their first year, (2) all TFA corps members in their first and second year of teaching, and (3) TFA alumni with a minimum two years of experience. Teacher experience was used as a balancing covariate during the PSW procedure for the alumni analyses only. The LR method is preferred for the estimation of the propensity scores, which served as a basis for the calculation of ATT weights. There are various methods used for the propensity score estimation, including complex techniques that are based on machine learning algorithms. Based on our experiences and some reported research results (Setoguchi et al., 2008), LR is thought to be a conventional yet powerful enough method for the propensity score estimations. CORE and TFA collaboratively identified a set of potentially important covariates to include in the PSW procedure. The contributing conditions or factors that were used for comparison group matching were: • student level: o gender o race o economic disadvantage o LEP o special education status o bilingual status o ESL status o being “at risk” according to the TEA 3 • teacher level: years of experience • school level: o percentage of the economically disadvantaged students in school o campus accountability rating (met standard or not) o charter status (traditional ISD/charter) • district level: accountability rating (met standard or not) PSW was performed for each analysis condition prior to the main LR model that predicted the TFA impact. Thus, the covariates included during each of the PSW analyses differed depending on the selected conditions, such as region, grade level, academic year. This approach is thought to improve the covariate balance for each of the selected subsamples, before the main LR model analyses. It is also important to note that CORE conducted some disaggregation analyses using student demographic characteristics. For these specific analyses, we excluded the demographic variable TEA At Risk Indicator Code indicates whether a student is currently identified as at-risk of dropping out of school using state-defined criteria only (TEC §29.081, Compensatory and Accelerated Instruction). 3 10 of interest from the PSW procedure. Sample sizes for each analysis were reported in terms of general, students taught by a TFA and non-TFA-affiliated teacher. After fitting the main LR model along with the estimated weight scores, CORE obtained the degree of the difference between the groups of students taught by a TFA and non-TFA-affiliated teacher in terms of the predicted passing probabilities. The statistical significance of this difference, as well as the magnitude of the TFA effect in terms of odds ratio, were reported for each of the analyses. These effect sizes were categorized based on some recommended thresholds (Cohen, 1988; Haddock et al., 1998; Lipsey & Wilson, 1993) in order to reflect the magnitude of the TFA effect as small, medium, and large. Nevertheless, it is important to note that such categorization depends on the research question and context. Thus, a small effect size in terms of odds ratios still can be considered an important effect, if the probability of pass is statistically higher in favor of students taught by a TFA-affiliated teacher. It is also important to note that some analysis results for specific conditions were not reported due to non-existing sub-samples and perfect probability levels (0/1) for either groups of students taught by a TFA or non-TFA-affiliated teacher. Moreover, CORE also excluded the analyses results where the sample sizes of students taught by a TFAaffiliated teacher were lower than 100. After these exclusions, a total of 699 different comparison analyses results were reported. Saturation CORE also explored the potential influence of “saturation” of TFA corps member and alumni teachers in a given context, hypothesizing that there could be setting-level effects of clustering TFA corps members and/or alumni that indirectly influence student outcomes. All analyses and results related to exploring the effects of saturation are provided in Appendix B. 11 Results Impact of TFA Key Takeaways • Overall, students of TFA corps members and alumni are as likely or more likely than students of non-TFA-affiliated teachers to pass STAAR in the year that they have the TFA-affiliated teacher. • Student Demographics: o The advantage of having a TFA-affiliated teacher is the same for economically disadvantaged and noneconomically disadvantaged students. o There is an advantage of TFA for students of all races; the effect is strongest specifically for Black and Hispanic students who have a TFA alumni as their teacher. o There is an advantage of TFA for both LEP and non-LEP students; the effect is strongest for LEP students. • Content Area: o There is an advantage of having a TFA-affiliated teacher across all content areas. o The largest advantage of TFA was in high school subjects. o In elementary and middle school, the relative advantage of TFA is greater for Math than Reading (TFA and nonTFA-affiliated teachers were nearly equally effective for Reading). • School Type: o There is an advantage of TFA in both traditional public ISD schools and charters; the effect is strongest in ISDs. o There is an advantage of TFA in schools that meet and do not meet accountability standards; the effect is strongest in school districts with low state accountability ratings. • Teacher Experience: o Findings differed by TFA affiliation (new corps members compared to corps members compared to alumni); the advantage of TFA is strongest with TFA alumni. • Findings differed by region. • Findings did not differ by grade level alone. Layout of Results Section This section describes a synthesis of all analyses and findings by specific content areas. For each, CORE utilized two main analytic strategies. 1. First, we describe the average estimated probability of passing STAAR for TFA and non-TFA conditions. 2. Second, to aid with interpretation of a large number of findings, we used a meta-analysis strategy to organize results into categories based on the statistical significance and direction of the differences (i.e., which group had a higher probability of passing) and report on those categories themselves (see Table Four). With this, we can describe how many individual analyses had findings represented by each of the 4 categories. Table Four. Legend: Categories of direction and significance of TFA/non-TFA analyses Finding category 4 3 2 1 Students of TFA-affiliated teachers are more likely to pass STAAR yes yes no no Result is statistically significant yes no no yes Additionally, findings are disaggregated by geographic region and supplemental explanation provided, as appropriate and available (e.g., for a given finding, we might explain differences between charters and ISDs). 12 Synthesis of All Analyses TFA vs non-TFA Probability of Passing STAAR In order to reduce a large number of analyses and findings, CORE adopted a meta-analysis strategy of pooling all analyses performed across grade levels, regions, and available years of data for a single content area to create a synthesized view of the overall effectiveness of TFA-affiliated teachers compared to non-TFA peers. Specifically, we averaged the probability of passing STAAR and compared students taught by a TFA-affiliated or non-TFA teacher. Analyses indicate that, overall, students of TFA corps member or alumni teachers have an equal or higher probability of passing the STAAR test in the year that they had that teacher compared to students of non-TFA-affiliated novice or veteran teachers. This overarching finding was achieved by synthesizing numerous comparisons in three categories: TFA alumni (225 total analyses), first-year TFA corps members (231 analyses), and all TFA corps members (243 analyses). In each instance, the TFA condition was compared to a non-TFA condition (i.e., the student taught by a TFA-affiliated teacher or not). These 699 analyses also applied to unique subsamples based on various combinations of factors including content area assessed (e.g., Reading or Math), grade level, geographic area, school year, school charter status, and student factors such as ethnicity, economically disadvantaged, and English proficiency status. Each comparison provided a probability of passing the STAAR assessment for students taught by a TFA-affiliated teacher in that year of the assessment and for those students taught by non-TFA-affiliated teachers, with the probability ranging from 0% predicted probability of passing to 100% probability. Figure One describes the overall average probability of passing the STAAR test in each content area for TFA-affiliated and non-TFA-affiliated teachers. This Figure pools estimated passing probabilities from all analyses conducted with two of the groups, all TFA corps members (n=243) and alumni (n=225), including all tested grade levels, all included regions, and all available years of data; the x-axis labels indicate the number of analyses represented by the content area. This aggregation does not include the findings from analyses conducted on just first year corps members. In all content areas, students of TFA corps member or alumni teachers were as likely as or more likely to pass STAAR than students of comparable teachers with no TFA affiliation 4. Figure One. Overall average probability of passing STAAR for all subjects; TFA-affiliated teachers compared to non-TFA peers Average Probability of Passing STAAR by Content Area Algebra 1 (n=43*) Biology (n=42) English 1 (n=38) English 2 (n=33) Math (n=103) Reading (n=104) Science (n=47) Social Studies (n=40) 41.6% 38.1% 28.9% 26.6% 39.9% 34.8% 39.2% 60.4% Avg. TFA Passing Probability 38.3% 37.4% 34.1% 55.8% 50.3% 45.8% 43.9% 52.4% 45.7% 34.8% 30.8% Avg. Non-TFA Passing Probability 69.2% Average finding of analyses of alumni (n=225 analyses) and all TFA corps members (n=243 analyses) US History All Subjects (n=18) Combined (n=468) *The N for each content area represents the number of analyses conducted for that content area; findings for these analyses are averaged to generate the average probability predicted by all content-specific analyses. 4 The passing probabilities are estimated based on parameters of the sample of students and teachers included in the analysis. These are not actual observed averages and will be necessarily different from actual passing percentages for the state of Texas at-large. Additionally, this sample is not representative of the population of the entire state of Texas, but rather of schools where TFA corps members and alumni taught during the years included in the sample. The average estimated probabilities are derived from multiple years and grade-levels. The average estimated probabilities should be interpreted conceptually, not as literal passing rates of groups. 13 Factors Associated with Differentiated TFA Effects In order to better understand how the estimated TFA effect changes across various conditions such as grade level and region, ANOVA was conducted as a simple meta-analysis of the 699 different comparison analyses. In this metaanalysis, each of the 699 individual impact analyses was an observation in the data. The dependent variable was the effect size of each analysis. The independent variables were the factors describing the focus of each analysis, such as grade level(s) included, geographic area(s), and STAAR subject(s). This allowed us to determine which conditions significantly predicted the effect of TFA, in other words, how the estimated TFA effects varied across analysis conditions. Upcoming sections of this report describe the details of these differentiated effects. The findings significantly varied by (see Appendix C): • TFA affiliation (new corps members, all corps members, alumni): The effect of TFA was different for TFA alumni than for active corps members. • The geographic region: The effect of TFA was different in the five regions of the state. • Subject: The effect of TFA was different for the various STAAR subjects, such as Reading & Math. • Note: The effect of TFA did not differ by grade level alone. Categorizing All Findings The result of each of the 699 unique analyses was categorized into one of four groups indicating both (1) which group (TFA or comparison) had a higher probability of passing, and (2) the statistical significance of the finding (see Table Five for legend). Figure Two presents 3rd – 8th grade Reading and Math comparison analyses categorized by “direction” and significance of findings. These findings are summarized for other content areas later in this section of the report. Students of TFA alumni were significantly more likely to pass STAAR Reading and Math in 77% and 82% of all Reading & Math analyses, respectively. TFA corps members are more effective in Math than Reading. Table Five. Categories of direction and significance of TFA/non-TFA analyses Finding category 4 3 2 1 Students of TFA-affiliated teachers are more likely to pass STAAR yes yes no no Result is statistically significant yes no no yes Figure Two. Reading and Math analyses by direction & significance findings categories Distribution of Reading & Math Comparison Findings (n=307 total analyses) Category 4 46% 44% 14% 12% 24% 16% Reading (n=52*) 18% 26% Math (n=52) Corps Members in 1st Year Category 3 Category 2 44% 60% Category 1 4% 10% 10% 12% 15% 23% 17% 4% 6% 8% 77% 82% Reading (n=52) Math (n=51) 21% 8% Reading (n=50) Math (n=50) All Corps Members (1st & 2nd year) TFA Alumni (2+ years) *The N for each group and content area represents the number of analyses conducted for that content area and group; the bar represents how the findings from that group of analyses are distributed within the four categories of possible findings. 14 Figure Three provides a heat map of the disaggregated comparison analyses by subject, geographic region, and TFA affiliation (alumni, corps member, etc.). An additional heat map organized by grade level is available in Appendix D. Using the same “category-of-finding” logic described previously, this heat map provides a comprehensive snapshot of which TFA-affiliated teachers are most effective for specific content areas within five distinct Texas regions. As can be seen from this map, TFA alumni are generally more effective than non-TFA-affiliated peer teachers across all regions (as indicated by mostly blue indicators for that group). Figure Three. Comparison analyses heat map disaggregated by TFA affiliation, geographic region, and content area 5 Region A Region B Region D Region C Region E Magnitude of Difference Based on these findings, it is critical to also discuss magnitude of difference. As can be seen in the heat map, new non-TFA-affiliated teachers are significantly more effective than their TFA corps member peers in several content areas and geographic regions. However, equally as important as statistical significance is the true magnitude of the difference, particularly with such a large sample size. Table Six describes the probability of passing STAAR for all subjects for the three groups of TFA-affiliated teachers and their comparison peer teachers (the three groups being brand new teachers, 0 years; new teachers in first or second year, and veteran teachers in their 3+ year of teaching). Across the three groups, the majority of time, the difference between TFA and non-TFA favors TFA. Table Six. Overall probability of passing STAAR for all subjects by teacher group 5 Heatmap legend: CM=Corps Member, US=US History, R=Reading, M=Math, E2=English 2, E1=English 1, BI=Biology, A1=Algebra 1. 15 Brand New Teachers in 1st Year Non-TFA TFA (new Corps Members) Difference (TFA-non) ALGEBRA 1 25.88% 26.23% BIOLOGY 38.78% ENGLISH 1 New Teachers in 1st or 2nd Year Veteran Teachers in 3rd+ Year Non-TFA TFA (all Corps Members) Difference (TFA-non) Non-TFA TFA (alumni) Difference (TFA-non) 0.35% 28.4% 27.8% -0.53% 33.4% 42.2% 8.8% 40.56% 1.78% 43.1% 45.1% 1.98% 48.5% 60.4% 11.9% 38.74% 38.06% -0.68% 39.0% 39.3% 0.37% 48.3% 51.6% 3.3% ENGLISH 2 43.99% 46.36% 2.37% 45.6% 47.3% 1.68% 55.9% 66.1% 10.2% MATH 31.48% 30.47% -1.01% 32.5% 31.5% -0.93% 35.7% 43.4% 7.7% READING 35.96% 33.83% -2.13% 37.1% 33.4% -3.73% 39.5% 45.0% 5.5% SCIENCE 27.38% 30.27% 2.89% 30.1% 32.0% 1.93% 39.4% 47.5% 8.2% SOCIAL STUDIES 23.46% 23.55% 0.10% 23.6% 23.9% 0.24% 30.0% 34.5% 4.5% U.S. HISTORY 63.48% 74.79% 11.31% 58.7% 68.3% 9.53% 68.5% 73.7% 5.2% ALL SUBJECTS 34.15% 34.49% 0.34% 36.1% 36.2% 0.08% 40.3% 47.5% 7.2% Student and School Subgroups Table Seven describes the differentiated effect of TFA corps member and alumni teachers on specific demographic subgroups of the population, such as student race and economically disadvantaged, and on specific groups of campuses such as charter schools compared to traditional ISD campuses. A table of detailed subgroup findings is available in Appendix E. Table Seven. Summary of overall trends observed for student subgroups Charter schools and traditional ISD schools Student race Economically disadvantaged Limited English proficient (LEP) TFA-affiliated teachers are more effective than non-TFA-affiliated teachers, on average, in both traditional ISD campuses and charter schools; this advantage over non-TFA-affiliated teachers is greater in ISD campuses than charters. In traditional ISDs, students of TFA alumni were 7% more likely to pass STAAR than students of nonTFA veteran teachers, compared to a 2% advantage in charter schools. Differences are less notable for TFA corps members; students of TFA corps members are slightly less likely to pass STAAR in charter schools (-0.8%), and slightly more likely to pass in traditional ISDs (0.2%). TFA-affiliated teachers are more effective than non-TFA-affiliated teachers, on average, for all ethnicities of students; this advantage over non-TFA-affiliated teachers is slightly greater and more consistent for Black and Hispanic students of TFA alumni. Black students of TFA alumni were 7.6% more likely to pass STAAR than black students of non-TFA veteran teachers, while Hispanic students of TFA alumni were 6.6% more likely to pass STAAR. This advantage of TFA alumni teachers was smaller for White students at 3.2%. However, for TFA corps members, there is not a notable advantage for Black and Hispanic students; Black students of corps members are 0.2% less likely to pass STAAR, and Hispanic students of corps members are 0.4% more likely to pass. TFA-affiliated teachers are equally effective, on average, for both economically disadvantaged (EcoDis) and non-economically disadvantaged students. EcoDis students of TFA-affiliated teachers are 2.3% more likely to pass STAAR than peers taught by non-TFA-affiliated teachers. This advantage is similar for non-EcoDis students with a 2.6% advantage for non-EcoDis students of TFA-affiliated teachers. This advantage is greater for students of TFA alumni; EcoDis students of TFA alumni are 6.7% more likely to pass STAAR, on average, and non-EcoDis students are 7.6% more likely to pass. Differences are less notable for TFA corps members; EcoDis students of TFA corps members are 0.4% more likely to pass STAAR, while non-EcoDis students of TFA corps members are 0.7% more likely to pass. TFA-affiliated teachers are more effective than non-TFA-affiliated teachers, on average, for both LEP and nonLEP students; this advantage over non-TFA-affiliated teachers is greater for LEP students. LEP students of TFAaffiliated teachers are 5.3% more likely to pass STAAR than peers taught by non-TFA-affiliated teachers. This advantage is 1.8% for non-LEP students. This advantage is greater for LEP students of TFA alumni; LEP students of TFA alumni are 9.3% more likely to pass STAAR, on average, compared to a 6.3% advantage for non-LEP students of TFA alumni. There is no relative advantage of TFA corps members for non-LEP students (-0.2% advantage), while LEP students of TFA corps members are 3.3% more likely to pass, on average. 16 TFA-affiliated teachers are more effective than non-TFA-affiliated teachers, on average, at both schools meeting state standards and those that are not meeting standard; this advantage over non-TFA-affiliated teachers is greater at schools not meeting standards. Students at schools not meeting state standards are 4.4% more likely to pass STAAR, on average, if they have a TFA-affiliated teacher. This advantage at schools meeting standards is 1.9%. This advantage is greater for students of TFA alumni. Students of TFA alumni at schools not meeting standards are 11.3% more likely to pass STAAR than students of non-TFA veteran teachers at similar schools. This advantage of TFA alumni at schools meeting standards is smaller, yet still notable, at 6.3%. There is no relative advantage of TFA corps members at schools that meet or do not meet standards. At schools not meeting standards, students of TFA corps members are 1% more likely to pass, while at schools that are meeting standards, students of TFA corps members are -0.1% less likely to pass STAAR. Campus performance Reading Using the same strategy described in the preceding section, CORE looked specifically at STAAR Reading scores. Reading findings represent all students in grades 3 through 8 in the five studied sites. Of all content areas explored, the largest difference in likelihood to pass STAAR that favored the non-TFA group was found in Reading. Students of TFA corps members are 3.7% less likely to pass STAAR Reading than students of new non-TFA-affiliated teachers (this difference is 2.1% favoring brand new non-TFA-affiliated teachers). Students of TFA alumni are 5.5% more likely to pass STAAR Reading compared to students taught by non-TFA-affiliated peers. A total of 154 individual analyses were conducted specifically for STAAR Reading for students in grades three through eight (TFA n=72,821 student scores; non-TFA n=2,757,931 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach as in the previous section, CORE organized Reading findings into categories (see Figure Five). Category 4 represents comparison analyses that favored the group of students taught by a TFA-affiliated teacher and was statistically significant. Category 1 represents analyses that favored the group of students taught by a non-TFA-affiliated teacher and was statistically significant. Category 2 represents non-significant findings representing non-TFA and Category 3 represents non-significant findings favoring TFA. (See Table Four). Figure Four describes the average probability of passing the Reading STAAR assessment for students of TFAaffiliated and non-TFA-affiliated teachers, disaggregated by both geographic area and grade level. Figure Five synthesizes all Reading comparison analyses categorized by “direction” and significance of findings. Table Eight describes observed trends in Reading for different subgroups of the study population. Figure Four. Average probability of passing STAAR Reading; TFA-affiliated teachers compared to non-TFA-affiliated peers Average Probability of Passing STAAR Reading by Geo Area Average Probability of Passing STAAR Reading by Grade Level Region RGV D (n=12) Region E San Antonio (n=13) 17 43.7% 47.4% 39.2% 37.7% Avg. TFA Passing Probability 35.3% 36.4% 40.6% 40.7% 28.5% 30.6% 46.5% Avg. Non-TFA Passing Probability 33.3% 33.2% Region HoustonC (n=14) 45.3% 40.0% 40.1% 40.4% Region DFW B (n=14) (all areas, all years, analyses of all TFA Corps Members and alumni) 32.7% 35.4% Region AustinA (n=4) 36.3% 44.8% 37.1% (all grade levels, all years, analyses of all TFA Corps Members and alumni) Avg. Non-TFA Passing Probability Avg. TFA Passing Probability 3rd (n=9) 4th (n=8) 5th (n=10) 6th (n=11) 7th (n=11) 8th (n=11) Figure Five. Reading analyses into direction & significance findings categories by age level; n=154 analyses Distribution of Reading Comparison Findings by Grade Level Category 4 23% 41% 23% Category 3 38% 27% 7% 7% 5% 5% 6% 11% 11% 40% 80% 13% 18% 13% ES (n=13) MS (n=15) All (n=22) ES (n=15) 68% 17% 91% 9% 20% 15% Corps Members in 1st Year Category 1 25% 27% 73% 14% Category 2 13% 72% 58% 18% 7% 5% MS (n=15) All (n=22) All Corps Members (1st & 2nd year) ES (n=12) MS (n=18) All (n=22) TFA Alumni (2+ years) Note: “ES” includes all analyses conducted exclusively for elementary grades (3-5), “MS” includes all analyses conducted exclusively for middle grades (6-8), and “All” includes the analyses that included all Reading-tested grade levels combined into a single sample. Table Eight. Differentiated effect of TFA teachers (corps members & alumni) on Reading achievement for demographic subgroups Probability of passing STAAR Reading with TFAaffiliated teacher 45.3% Probability of passing STAAR Reading with non-TFA-affiliated teacher 51.3% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD 35.3% 34.2% Student race Black 33.9% 33.8% Hispanic 38.6% 38.3% White 64.8% 60.0% Economically disadvantaged Yes 36.9% 36.4% No 60.7% 61.9% Limited English proficient Yes 25.8% 23.5% No 43.8% 44.3% Met standard 41.6% 41.6% Did not meet 24.4% 23.4% Campus performance Math Next, CORE looked specifically at STAAR Math outcomes. Math findings represent students in grades 3 through 8 in all five studied sites. Students of TFA corps members are 0.9% less likely to pass STAAR Math than students of new non-TFA-affiliated teachers (this difference is 1% favoring brand new non-TFA-affiliated teachers). Students of TFA alumni are 7.7% more likely to pass STAAR Math compared to students taught by non-TFA-affiliated peers. A total of 153 individual analyses were conducted specifically for STAAR Math for students in grades three through eight (TFA n=81,188 student scores; non-TFA n=3,495,767 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized Math findings into categories. Category 4 represents comparison analyses that favored the group of students taught by a TFA-affiliated teacher and was statistically significant. Category 1 represents analyses that favored the group of students taught by a 18 non-TFA-affiliated teacher and was statistically significant. Category 2 represents non-significant findings representing non-TFA and Category 3 represents non-significant findings favoring TFA. (See Table Four for legend). Figure Six describes the average probability of passing the Math STAAR assessment for students of TFA-affiliated and non-TFA-affiliated teachers, disaggregated by both geographic area and grade level. Figure Seven synthesizes all Math comparison analyses categorized by “direction” and significance of findings. Table Nine describes observed trends in Reading for different subgroups of the study population. Figure Six. Average probability of passing STAAR Math; TFA-affiliated teachers compared to non-TFA-affiliated peers Average Probability of Passing STAAR Math by Geo Area Average Probability of Passing STAAR Math by Grade Level (all grade levels, all years, TFA Corps Members and alumni) Region RGV D (n=11) Region E San Antonio (n=14) 5th (n=10) 6th (n=11) 7th (n=10) 26.5% 27.5% 30.6% 36.3% 4th (n=9) Avg. TFA Passing Probability 37.5% 44.5% 3rd (n=9) 44.6% 46.8% 31.0% 27.4% 39.4% 39.5% 40.1% 37.4% Region C Houston (n=14) Avg. Non-TFA Passing Probability 31.8% 33.6% Region DFWB (n=12) Avg. TFA Passing Probability 31.3% 36.6% Region AustinA (n=4) 39.8% 26.9% 34.6% 45.6% Avg. Non-TFA Passing Probability (all areas, all years, TFA Corps Members and alumni) 8th (n=9) Figure Seven. Math analyses organized by direction & significance findings categories; n=153 analyses Distribution of Math Comparison Findings by Grade Level Category 4 23% 23% 53% 13% 5% Category 2 47% 7% MS (n=15) 45% 33% 23% 20% 20% 23% All (n=22) ES (n=15) MS (n=15) All (n=22) 27% Corps Members in 1st Year Category 1 15% 13% 15% 7% 7% 4% 96% 18% 23% 20% 13% ES (n=13) 40% 7% 8% 46% 50% Category 3 14% All Corps Members (1st & 2nd year) 69% 73% ES (n=13) MS (n=15) All (n=23) TFA Alumni (2+ years) Note: “ES” includes all analyses conducted exclusively for elementary grades (3-5), “MS” includes all analyses conducted exclusively for middle grades (6-8), and “All” includes the analyses that included all Math-tested grade levels combined into a single sample. 19 Table Nine. Differentiated effect of TFA teachers (corps members & alumni) on Math achievement for demographic subgroups Probability of passing Math STAAR with TFA-affiliated teacher Probability of passing Math STAAR with nonTFA-affiliated teacher 43.4% Charter schools and traditional ISD schools Open enrollment charter 42.8% Traditional ISD 31.9% 30.2% Student race Black 28.2% 27.3% Hispanic 36.6% 34.2% White 53.8% 49.6% Economically disadvantaged Yes 34.9% 32.7% No 47.9% 44.2% Limited English proficient Yes 31.1% 27.1% No 38.0% 36.3% Campus performance Met standard 38.4% 36.6% Did not meet 24.2% 20.3% Science Next, CORE looked specifically at STAAR Science outcomes. These findings represent students in grades 5 and 8 in all studied sites. Students of TFA corps members are 1.9% more likely to pass STAAR Science than students of new non-TFA-affiliated teachers (this difference is 2.9% favoring brand new TFA-affiliated teachers). Students of TFA alumni are 8.2% more likely to pass STAAR Science compared to students taught by non-TFA-affiliated peers. A total of 70 individual analyses were conducted specifically for STAAR Science for students in 5th and 8th grades (TFA n=29,244 student scores; non-TFA n=1,044,581 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized Science findings into categories. Category 4 represents comparison analyses that favored the group of students taught by a TFA-affiliated teacher and was statistically significant. Category 1 represents analyses that favored the group of students taught by a non-TFAaffiliated teacher and was statistically significant. Category 2 represents non-significant findings representing non-TFA and Category 3 represents non-significant findings favoring TFA. (See Table Four for legend). Figure Eight describes the average probability of passing the Science STAAR assessment for students of TFAaffiliated and non-TFA-affiliated teachers, disaggregated by both geographic area and grade level. Figure Nine synthesizes all Science comparison analyses categorized by “direction” and significance of findings. Table Ten describes observed trends in Science for different subgroups of the study population. Figure Eight. Average probability of passing STAAR Science; TFA-affiliated teachers compared to non-TFA-affiliated peers Average Probability of Passing STAAR Science by Geo Area Average Probability of Passing STAAR Science by Grade Level (all grade levels, all years, TFA Corps Members and alumni) Avg. TFA Passing Probability Region HoustonC (n=2) Region RGV D (n=2) Region E San Antonio (n=2) 20 5th (n=2) 8th (n=2) 40.2% 34.9% 41.3% 37.6% 33.8% 32.5% 24.4% 46.6% 44.5% 40.0% 40.4% 36.4% Region DFW B (n=2) 29.7% Region AustinA (n=1) 28.5% 69.4% 47.6% Avg. Non-TFA Passing Probability (all areas, all years, TFA Corps Members and alumni) Avg. Non-TFA Passing Probability Avg. TFA Passing Probability Both (n=43) Figure Nine. Science analyses organized by direction & significance findings categories; n=70 analyses Distribution of Science Comparison Findings by Grade Level Category 4 Category 3 Category 2 5% 5% Category 1 5% 10% 5% 14% 29% 100% 100% 100% 100% 100% 100% 95% ES (n=1) MS (n=1) All (n=22) 76% 57% ES (n=1) MS (n=1) All (n=21) Corps Members in 1st Year ES (n=1) MS (n=1) All (n=21) All Corps Members (1st & 2nd year) TFA Alumni (2+ years) Note: “ES” includes all analyses conducted exclusively for elementary grades (3-5), “MS” includes all analyses conducted exclusively for middle grades (6-8), and “All” includes the analyses that included all Science-tested grade levels combined into a single sample. Table Ten. Differentiated effect of TFA teachers (corps members & alumni) on Science achievement for demographic subgroups Probability of passing Science STAAR with TFAaffiliated teacher 50.6% Probability of passing Science STAAR with non-TFA-affiliated teacher 47.4% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD 35.8% 30.9% Student race Black 32.5% 28.6% Hispanic 40.3% 36.4% n/a n/a White Economically disadvantaged Yes 38.4% 34.3% No 56.8% 52.5% Limited English proficient Yes 25.3% 21.3% No 44.6% 40.4% Met standard 42.8% 39.4% Did not meet 24.8% 17.9% Campus performance n/a = no valid individual analyses applied to that specific sub-group for Science Social Studies Next, CORE looked specifically at STAAR Social Studies outcomes for 8th grade students. Students of TFA corps members are 0.2% more likely to pass STAAR Social Studies than students of new non-TFA-affiliated teachers (this difference is just 0.1% favoring brand new TFA-affiliated teachers). Students of TFA alumni are 4.5% more likely to pass STAAR Social Studies compared to students taught by non-TFA-affiliated peers. A total of 61 individual analyses were conducted specifically for STAAR Social Studies for students in 8th grade students (TFA n=19,592 student scores; non-TFA n=848,502 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized Social Studies findings into categories. Category 4 represents comparison analyses that favored the group of students taught by a TFAaffiliated teacher and was statistically significant. Category 1 represents analyses that favored the group of students by 21 a non-TFA-affiliated teacher and was statistically significant. Category 2 represents non-significant findings representing non-TFA and Category 3 represents non-significant findings favoring TFA. (See Table Four for legend). Figure Ten describes the average probability of passing the Social Studies STAAR assessment for students of TFAaffiliated and non-TFA-affiliated teachers, disaggregated by geographic area (grade level disaggregation is not applicable as Social Studies is taken by 8th grade students only). Figure 11 synthesizes all Social Studies comparison analyses categorized by “direction” and significance of findings. Table 11 describes observed trends in Social Studies for different subgroups of the study population. Figure Ten. Average probability of passing STAAR SocStud Average Probability of Passing STAAR Social Studies by Geo Area (all grade levels, all years, TFA Corps Members and alumni) Figure 11. Social studies analyses organized by direction & significance findings categories; n=61 analyses Distribution of Social Studies Comparison Findings Category 4 Category 3 Category 2 Category 1 10.53% 28.57% Avg. Non-TFA Passing Probability Avg. TFA Passing Probability 28.57% 14.29% 19.05% 28.57% 23.81% 19.05% Corps Members in 1st Year All Corps Members (1st & 2nd year) Region HoustonC (n=2) 14.7% 11.2% 27.7% 26.2% Region DFW B (n=2) 34.2% 36.7% 21.5% 26.2% 60.5% 28.6% Region AustinA (n=1) Region RGV D (n=2) Region E San Antonio (n=1) 38.10% 21.05% 10.53% 57.89% TFA Alumni (2+ years) Table 11. Differentiated effect of TFA teachers (corps members & alumni) on Social Studies achievement for demographic subgroups Probability of passing Social Studies STAAR with TFAaffiliated teacher 38.8% Probability of passing Social Studies STAAR with non-TFAaffiliated teacher 41.2% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD 21.5% 21.1% Student race Black 22.9% 21.6% Hispanic 27.1% 28.4% n/a n/a White Economically disadvantaged Yes 25.4% 26.9% No 46.4% 41.1% Limited English proficient Yes 12.7% 13.7% No 30.8% 31.7% Met standard 29.8% 31.4% Did not meet 13.8% 9.7% Campus performance n/a = no valid individual analyses applied to that specific sub-group for Social Studies 22 High School End of Course (EOC) Exams Algebra 1 Next, CORE looked specifically at STAAR Algebra 1 outcomes. Students of TFA corps members are 0.5% less likely to pass the Algebra 1 EOC exam than students of new non-TFA-affiliated teachers (this difference favors brand new TFA-affiliated teachers by 0.4%). Students of TFA alumni are 8.8% more likely to pass the Algebra 1 exam compared to students taught by non-TFA-affiliated peers. A total of 64 individual analyses were conducted specifically for Algebra 1 for students in grades 8-12 (TFA n=23,841 student scores; non-TFA n=625,757 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized Algebra 1 findings into categories. Figure 12 describes the average probability of passing the Algebra 1 EOC exam for students of TFA-affiliated and non-TFA-affiliated teachers, disaggregated by geographic area. Figure 13 synthesizes all Algebra 1 comparison analyses categorized by “direction” and significance of findings. Table 12 describes observed trends in Algebra 1 for different subgroups of the study population. Figure 12. Average probability of passing Algebra 1 EOC Figure 13. Algebra 1 analyses organized by direction & significance findings categories; n=64 analyses Distribution of Algebra 1 Comparison Findings Category 4 Category 3 Category 2 Category 1 Average Probability of Passing STAAR Algebra 1 by Geo Area (all grade levels, all years, TFA Corps Members and alumni) 9.52% 19.05% Avg. Non-TFA Passing Probability Avg. TFA Passing Probability 40.91% 19.05% 28.9% 39.6% 37.3% 36.6% 38.0% 40.6% Region AustinA (n=1) Region DFW B (n=2) Region HoustonC (n=2) Region RGV D (n=2) 25.8% 18.2% 31.0% 35.6% 14.29% 47.62% Region E San Antonio (n=1) Corps Members in 1st Year 13.64% 90.48% 22.73% 22.73% All Corps Members (1st & 2nd year) TFA Alumni (2+ years) Table 12. Differentiated effect of TFA teachers (corps members & alumni) on Algebra 1 EOC exam for demographic subgroups Probability of passing Algebra 1 EOC with TFA-affiliated teacher 37.9% 34.5% Probability of passing Algebra 1 EOC with non-TFA-affiliated teacher 40.9% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD Student race Black 27.6% 21.8% Hispanic White 35.6% 46.2% 32.5% 38.4% Economically disadvantaged Yes 33.4% 29.9% No 48.1% 41.2% Limited English proficient Yes 34.2% 25.7% No 35.5% 32.9% Campus performance Met standard 38.4% 35.3% Did not meet 18.1% 14.2% 23 30.6% Biology Next, CORE looked specifically at STAAR Biology outcomes. Students of TFA corps members are 1.9% more likely to pass the Biology EOC exam than students of new non-TFA-affiliated teachers (this difference favors brand new TFA-affiliated teachers by 1.8%). Students of TFA alumni are 11.9% more likely to pass the Biology exam compared to students taught by non-TFA-affiliated peers. A total of 64 individual analyses were conducted specifically for Biology for students in grades 9-12 (TFA n=27,869 student scores; non-TFA n=840,456 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized Biology findings into categories. Figure 14 describes the average probability of passing the Biology EOC exam for students of TFA-affiliated and nonTFA-affiliated teachers, disaggregated by geographic area. Figure 15 synthesizes all Biology comparison analyses categorized by “direction” and significance of findings. Table 13 describes observed trends in Biology for different subgroups of the study population. Figure 14. Average probability of passing Biology EOC Figure 15. Biology analyses organized by direction & significance findings categories; n=64 analyses Distribution of Biology Comparison Findings Category 4 Category 3 Category 2 Category 1 Average Probability of Passing STAAR Biology by Geo Area (all grade levels, all years, TFA Corps Members and alumni) Region DFW B (n=2) Region HoustonC (n=2) Region RGV D (n=2) 10.00% 27.27% 18.18% 13.64% 27.27% 90.00% 34.5% 39.3% 51.8% 57.8% 55.0% 51.7% 45.7% 60.9% 54.5% 58.6% Avg. Non-TFA Passing Probability Avg. TFA Passing Probability Region AustinA (n=1) 4.55% 13.64% Region E San Antonio (n=2) 54.55% 40.91% Corps Members in 1st Year All Corps Members (1st & 2nd year) TFA Alumni (2+ years) Table 13. Differentiated effect of TFA teachers (corps members & alumni) on Biology EOC exam achievement for demographic subgroups Probability of passing Biology EOC with TFA-affiliated teacher 68.6% Probability of passing Biology EOC with non-TFA-affiliated teacher 69.2% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD 51.8% 43.8% Student race Black 47.7% 39.0% Hispanic 53.5% 47.6% White 63.2% 65.6% Economically disadvantaged Yes 51.3% 44.7% No 52.8% 53.3% Limited English proficient Yes 40.9% 29.2% No 57.3% 51.6% Campus performance Met standard 56.6% 51.5% Did not meet 41.1% 24.2% 24 English 1 Next, CORE looked specifically at STAAR English 1 outcomes. Students of TFA corps members are 0.4% more likely to pass the English 1 EOC exam than students of new non-TFA-affiliated teachers (this difference favors brand new non-TFA-affiliated teachers by 0.7%). Students of TFA alumni are 3.3% more likely to pass the English 1 exam compared to students taught by non-TFA-affiliated peers. A total of 56 individual analyses were conducted specifically for English 1 for students in grades 9-12 (TFA n=10,954 student scores; non-TFA n=548,308 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized English 1 findings into categories. Figure 16 describes the average probability of passing the English 1 EOC exam for students of TFA-affiliated and non-TFA-affiliated teachers, disaggregated by geographic area. Figure 17 synthesizes all English 1 comparison analyses categorized by “direction” and significance of findings. Table 14 describes observed trends in English 1 for different subgroups of the study population. Figure 16. Average probability of passing English 1 EOC Figure 17. English 1 analyses organized by direction & significance findings categories; n=56 analyses Distribution of English 1 Comparison Findings Category 4 Category 3 Category 2 Category 1 Average Probability of Passing STAAR English 1 by Geo Area (all grade levels, all years, TFA Corps Members and alumni) Region DFW B (n=2) Region HoustonC (n=2) Region RGV D (n=2) 33.33% 25.00% 15.00% 33.33% 41.6% 44.9% 56.7% 57.5% 48.9% 52.0% 37.9% 38.0% 34.9% 34.2% Region AustinA (n=1) 11.11% 27.78% Avg. Non-TFA Passing Probability Avg. TFA Passing Probability 38.89% 27.78% Region E San Antonio (n=2) 11.11% 16.67% Corps Members in 1st Year All Corps Members (1st & 2nd year) 60.00% TFA Alumni (2+ years) Table 14. Differentiated effect of TFA teachers (corps members & alumni) on English 1 EOC exam for demographic subgroups Probability of passing English 1 EOC with TFA-affiliated teacher 64.9% Probability of passing English 1 EOC with non-TFA-affiliated teacher 59.9% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD 38.8% 38.4% Student race Black 35.7% 37.2% Hispanic 47.3% 44.6% White 67.0% 72.6% Economically disadvantaged Yes 43.3% 41.3% No 59.0% 57.3% Limited English proficient Yes 27.0% 19.9% No 50.1% 49.8% Campus performance Met standard 47.7% 46.1% Did not meet 28.2% 24.1% 25 English 2 Next, CORE looked specifically at STAAR English 2 outcomes. Students of TFA corps members are 1.7% more likely to pass the English 2 EOC exam than students of new non-TFA-affiliated teachers (this difference favors brand new TFA-affiliated teachers by 2.4%). Students of TFA alumni are 10.2% more likely to pass the English 2 exam compared to students taught by non-TFA-affiliated peers. A total of 51 individual analyses were conducted specifically for English 2 for students in grades 9-12 (TFA n=8,546 student scores; non-TFA n=498,543 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized English 2 findings into categories. Figure 18 describes the average probability of passing the English 2 EOC exam for students of TFA-affiliated and non-TFA-affiliated teachers, disaggregated by geographic area. Figure 19 synthesizes all English 2 comparison analyses categorized by “direction” and significance of findings. Table 15 describes observed trends in English 2 for different subgroups of the study population. Figure 18. Average probability of passing English 2 EOC Average Probability of Passing STAAR English 2 by Geo Area Figure 19. English 2 analyses organized by direction & significance findings categories; n=51 analyses Distribution of English 2 Comparison Findings Category 4 Category 3 Category 2 Category 1 (all grade levels, all years, TFA Corps Members and alumni) 5.56% 5.56% 11.11% 66.67% 50.00% 22.22% 27.78% Corps Members in 1st Year All Corps Members (1st & 2nd year) AustinA Region (n=1) DFW B Region (n=1) HoustonC Region (n=`) RGV D Region (n=1) 42.1% 54.9% 65.0% 68.2% 54.8% 56.8% 43.3% 49.5% 73.1% 69.9% Avg. Non-TFA Passing Probability Avg. TFA Passing Probability San Antonio Region E (n=1) 6.67% 11.11% 93.33% TFA Alumni (2+ years) Table 15. Differentiated effect of TFA teachers (corps members & alumni) on English 2 EOC exam for demographic subgroups Probability of passing English 2 EOC with TFA-affiliated teacher 75.9% Probability of passing English 2 EOC with non-TFA-affiliated teacher 73.0% Charter schools and traditional ISD schools Open enrollment charter Traditional ISD 54.8% 48.1% Student race Black 48.2% 42.4% Hispanic 61.6% 54.6% n/a n/a White Economically disadvantaged Yes 57.5% 50.4% No 56.5% 55.8% Limited English proficient Yes 27.6% 18.2% No 64.3% 58.4% Campus performance Met standard 62.5% 56.0% Did not meet 25.5% 21.1% n/a = no valid individual analyses applied to that specific sub-group for biology 26 United States History Finally, CORE looked specifically at STAAR U.S. History outcomes. Students of TFA corps members are 9.5% more likely to pass the U.S. History EOC exam than students of new non-TFA-affiliated teachers (this difference favors brand new TFA-affiliated teachers by 11.3%). Students of TFA alumni are 5.2% more likely to pass the U.S. History exam compared to students taught by non-TFA-affiliated peers. A total of 26 individual analyses were conducted specifically for U.S. History for students in grades 9-12 (TFA n=1,779 student scores; non-TFA n=140,956 student scores). The subsamples used in the individual analyses were differentiated based on the various factors previously described, such as school year, grade level, demographic subgroups, etc. Utilizing the same data-reduction approach, CORE organized U.S. History findings into categories. Figures 20 describes the average probability of passing the U.S. History EOC exam for students of TFA-affiliated and non-TFA-affiliated teachers, disaggregated by geographic area. Figure 21 synthesizes all U.S. History comparison analyses categorized by “direction” and significance of findings. Table 16 describes observed trends in U.S. History for different subgroups of the study population. Figure 20. Average probability of passing U.S. History EOC Figure 21. U.S. History analyses organized by direction & significance findings categories; n=26 analyses Distribution of U.S. History Comparison Findings Category 4 Category 3 Average Probability of Passing STAAR US History by Geo Area 25.00% 61.4% 77.6% 66.4% 71.3% 51.6% 59.5% (all grade levels, all years, TFA Corps Members and alumni) Avg. Non-TFA Passing Probability Avg. TFA Passing Probability 20.00% 66.67% 75.00% 80.00% 33.33% Region AustinA (n=0) Region DFW B (n=1) Region HoustonC (n=1) Region RGV D (n=1) Region E San Antonio (n=0) Corps Members in 1st Year All Corps Members (1st & 2nd year) TFA Alumni (2+ years) Table 16. Differentiated effect of TFA teachers (corps members & alumni) on U.S. History EOC exam for demographic subgroups Charter schools and traditional ISD schools Open enrollment charter Student race Black Probability of passing U.S. History EOC with TFA-affiliated teacher 83.6% Probability of passing U.S. History EOC with non-TFA-affiliated teacher 75.3% 55.6% 52.6% n/a n/a 72.4% 65.2% n/a n/a Traditional ISD Hispanic White Economically disadvantaged Yes 72.0% 65.2% No n/a n/a Limited English proficient Yes 41.5% 26.3% No 75.3% 66.7% Campus performance Met standard 72.8% 64.4% Did not meet n/a n/a n/a = no valid individual analyses applied to that specific sub-group for U.S. History 27 Discussion This evaluation provides a rigorous look at TFA impacts over multiple years and in its largest geographic region in the United States. This report addresses the initial question about TFA effects by asking: Are TFA teachers more or equally effective as matched non-TFA affiliated peer teachers? This evaluation examined student-level STAAR pass probabilities by propensity score weighting and logistic regression. Finally, meta-analyses were used to determine how estimated effects of TFA differed across various conditions. Similar to prior studies, this evaluation uses a quasiexperimental design and PSW procedure to carefully match students in the TFA condition with peers taught by a non-TFA-affiliated teacher in a given year (Hansen et al., 2015; Henry et al., 2015; Turner et al., 2012; Ware et al., 2011). An improvement on earlier quasi-experimental research on TFA, this study uses deidentified matched student and teacher-level data to isolate the impact of teachers on student achievement based on their affiliation with TFA. In addition, analyses include student and teacher data from every region in Texas, allowing for generalizability across the state, and include core STAAR tested subjects in elementary and secondary grade levels. This report also provides a preliminary analysis of the saturation question, probing the effects of clustering TFA-affiliated teachers in a given school setting (See Appendix B). Overall, across all content areas, students of TFA-affiliated teachers were as likely as or more likely to pass the Texas STAAR assessment than students of non-TFA-affiliated teachers. These findings have significant implications for public and private agencies seeking to continue or expand support for the TFA model especially has the relative cost and benefit of TFA’s alternative certification strategy is taken into account. These results varied across regions and student demographic subgroups; CORE has described in detail what these various conditions are and encourages TFA and other stakeholders to incorporate this knowledge into ongoing improvements and iterations of the overall TFA model. Importantly, the bulk of the benefit from TFA seems to come from alumni as opposed to corps members. This is no surprise given the on-the-job learning that occurs during any teacher’s first years in the classroom. This finding in particular serves to underscore the importance of retention. Strategic efforts to keep TFA alumni in classrooms is warranted as is ongoing study of what conditions catalyze long term teacher tenure. Limitations of the Study A notable limitation of the current study—though additional analyses to address this are already planned—is that previous achievement levels of the students were not controlled during the PSW procedure. Because the focus of this current study was Texas impact broadly, prior performance was not controlled for due to data-loss concerns. For instance, all 2011-12 school year and all 3rd grade data for six years would be excluded from that cohort view, as the records of prior achievement will not be available. Nevertheless, understanding the relative impacts of having a TFA corps member or alumni teacher in conjunction with all of the other covariates addressed in the PSW procedure provide actionable and meaningful outcome data. The current analyses predicted STAAR pass likelihood and controlled for a great deal of contributing variability but planned analyses will further strengthen evidence related to TFA impacts by also controlling for prior academic performance. This will require identification of a “good track” cohort of students who have sufficient data across multiple years. This cohort will be slightly smaller than the existing sample, although sample sizes should remain sufficient. The current analyses examine academic outcomes for quite traditional students. That is, for this evaluation we made an analytic decision to exclude: special education students, students who changed schools during the school year, and teachers who came in part way through a school year or who were not the teacher of record (e.g., teacher’s aide or assistant). Additional exclusions—that were warranted given the central evaluation questions for this report— included deciding to only analyze data for teachers who were the only teacher of record for a given school year. This necessarily excluded teachers who may have co-taught with other teachers. Though this evaluation sought to establish impacts for the typical student experience first, these excluded samples represent critically important variation in student and teacher experiences that should be taken into account in future planned analyses. Due to the smaller 28 sample sizes and infrequent occurrence, these analyses will require meaningfully different analyses than the current study employed and perhaps utilize mixed-methods approaches. While the current analyses identify differences in student outcomes that can be attributed to having a TFA-affiliated teacher or not, they do not tell us anything in particular about the non-TFA-affiliated group of teachers. Other than knowing the number of years the non-TFA-affiliated sample had been teaching, we do not have available data about their terminal degrees, whether they attended a traditional or alternative teacher certification program, or even whether they had teaching experience in another state. The nature of the data necessitated that we treat the non-TFA-affiliated group quite homogeneously while they certainly are not. Identifying sub-groups of non-TFA-affiliated teachers would have important implications for understanding TFA’s impact relative to other alternative teacher certifications programs. Future Research This report represents the first in a series of analyses that CORE and TFA have planned for these data (see Figure 22 for a summary of all planned analyses). Pending, and planned, analyses relate to looking at TFA effects over several years in a row. One is related to consistency, or the effects on student academic outcomes of having at least one TFA-affiliated teacher for multiple years in a row. Additionally, we seek to understand sustainability of effects of having a TFA-affiliated teacher for one or more years. All pending analyses will require identifying “good track” data – student level records that can be followed over multiple years, linking students to key outcome and covariate conditions. Figure 22: Full set of planned analysis questions Comparison Consistency Sustainability Saturation Comparing students taught by TFA and non-TFA affiliated teachers Effect of exposure to a TFA-affiliated teacher for multiple years in a row Lasting effects of exposure to TFA-affiliated teachers Impacts of being in a setting with multiple TFAaffiliated teachers Context: What factors explain variability in findings? 29 References Clark, M. A., Chiang, H. S., Silva, T., McConnell, S., Sonnenfeld, K., Erbe, A., & Puma, M. (2013). The effectiveness of secondary Math teachers from Teach For America and the Teaching Fellows programs (NCEE 2013-4015). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://eric.ed.gov/?q=Teach+for+America+&pg=2&id=ED544171 Clark, M. A., Isenberg, E., Liu, A. Y., Makowsky, L., & Zukiewicz, M. (2017). Impacts of the Teach for America Investing in Innovation scale-up. Princeton, NJ: Mathematica Policy Research. Retrieved from http://www. Mathematica-mpr.com/our-publications-and-findings/publications/impacts-of-the-teach-for-americainvesting-in-innovation-scaleup Cohen, J. E. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. Decker, P. T., Mayer, D. P., and Glazerman, S. (2004). The effects of Teach For America on students: Findings from a national evaluation. Mathematica Policy Research. Retrieved from www.Mathematica-mpr.com/ourpublications-and-findings Glazerman, S., Mayer, D., & Decker, P. (2006). Alternative routes to teaching: The impacts of Teach for America on student achievement and other outcomes. Journal of Policy Analysis and Management, 25(1), 75-96. Haddock, C., Rindskopf, D., & Shadish, W. (1998). Using odds ratios as effect sizes for meta-analysis of dichotomous data: A primer on methods and issues. Psychological Methods, 3(3), 339-353. Hansen, M., Backes, B., Brady, V., & Xu, Z. (2015). Examining spillover effects from Teach for America corps members in Miami–Dade County Public Schools (CALDER Working Paper 113). Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. Retrieved from https://caldercenter.org/sites/default/files/WP%20113_0.pdf Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15(3), 234-249. Henry, G. T., Purtell, K. M., Bastian, K. C., Fortner, C. K., Thompson, C. L., Campbell, S. L., & Patterson, K. M. (2014). The effects of teacher entry portals on student achievement. Journal of Teacher Education, 65(1), 723. Houston Independent School District. (n.d.). Teach for America (TFA) 2010−2011. Houston, TX: Houston Independent School District, Department of Research and Accountability. Retrieved from https://www.houstonisd.org/cms/lib2/TX01001591/Centricity/Domain/8269/TFA%202011full%20repor t.pdf Houston Independent School District. (2018, January). Teach for America evaluation, 2017. Houston, TX: Houston Independent School District, Department of Research and Accountability. Retrieved from https://www.houstonisd.org/cms/lib2/TX01001591/Centricity/domain/8269/pe_cirriculum/TFA%20Pro gram%20Eval%20Report_12%2018%2017-dy.pdf Leite, W. L. (2016). Practical propensity score methods using R. Thousand Oaks, CA: Sage. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage. Mickelson, N. R., & McEnturff, A. (2015, January). Evaluation of Teach for America: 2014-2015 (EA 15-536-2). Dallas, TX: Dallas Independent School District, Department of Evaluation and Assessment. Retrieved from 30 https://www.dallasisd.org/cms/lib/TX01001475/Centricity/domain/98/evaluation/14-15/finalrpts/EA15536-2%20Teach%20For%20America.pdf Penner, E. K. (2016). Teaching for all? Teach For America’s effects on the distribution of student achievement. Journal of Research on Educational Effectiveness, 9(3), 259-282. https://doi.org/10.1080/19345747.2016.1164779 Raymond, M., Fletcher, S. H., and Luque, J. (2001). Teach For America: An evaluation of teacher differences and student outcomes in Houston, Texas. Stanford, CA: Hoover Institution, CREDO. Retrieved from http://credo.stanford.edu/downloads/tfa.pdf Selya, A. S., Rose, J. S., Dierker, L. C., Hedeker, D., & Mermelstein, R. J. (2012). A practical guide to calculating Cohen’s f2, a measure of local effect size, from PROC MIXED. Frontiers in Psychology, 3, 111. https://doi.org/10.3389/fpsyg.2012.00111 Setoguchi, S., Schneeweiss, S., Brookhart, M. A., Glynn, R.J., & Cook, E. F. (2008). Evaluating uses of data mining techniques in propensity score estimation: A simulation study. Pharmacoepidemiology and Drug Safety, 17(6), 546-555. https://doi.org/10.1002/pds.1555 Turner, H. M., Goodman, D., Adachi, E., Brite, J., & Decker, L. E. (2012). Evaluation of Teach For America in Texas schools. San Antonio, TX: Edvance Research. U.S. Department of Education. (2016, August). Teach for America. What Works Clearinghouse Intervention Report. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc_tfa_083116.pdf Vasquez Heilig, J. & Jez, S. J. (2014). Teach for America: A Return to the Evidence. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/teach-for-america-return Ware, A., LaTurner, R. J., Parsons, J., Okulicz-Kozaryn, A., Garland, M., & Klopfenstein, K. (2011). Teacher preparation programs and Teach for America research study. Dallas, TX: Education Research Center, The University of Texas at Dallas. Xu, Z., Hannaway, J., & Taylor, C. (2007). Making a difference? The effects of Teach For America in high school (CALDER Working Paper 17). Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. Retrieved from: https://eric.ed.gov/?id=ED509654 31 Appendix A: Details of Data Selection The detailed information about the data cleaning procedure is given below. • Teachers who labelled as TFA-affiliated and non-TFA-affiliated within the same academic year were omitted. Number of the teachers who met that criteria were very small, min/max were 8/27 over the six academic years. • TFA corps members with more than two years of experience we omitted as they do not match the working definition of the experience and training of a typical corps members. A total of 13 teachers and their student data were excluded from all datasets. • Districts that did not have a TFA-affiliated teacher in none of the six academic years were omitted. 130 of the 257 districts were discarded from the analyses. 8.6% (n=16,037) of the teachers were discarded depending on this decision along with their student data. • Teachers that had missing observations for their roles (i.e., Support, Assistant, and Teacher of Record) were discarded and only teachers of record were kept for the analyses. Approximately 1% of the TFA-affiliated teachers were assistant or support teachers. Thus, data loss was not that excessive for TFA-affiliated teachers. • Students that changed their schools during an academic year were excluded from the datasets. Since there was not a school ID in the data files, we excluded all students that had multiple observations for the schoollevel teacher related information like head counts. Approximately 5% (n=85,000) of the unique students were discarded from each of the campus-level files for six years. • Students that took the alternate version of the STAAR and/or had a test accommodation indicator as Braille were excluded. The min/max number of the students excluded were 1.6% (n=27,077) / 4.6% (n=73,411) over the six academic years. • For the 3rd to 5th grade data, students who matched to more than a single teacher ID were excluded. The min/max data loss was 57% (n=258,132) / 72% (n=383,255) over the six academic years. For the 6th to 12th grade analyses, this exclusion based on a specific subject area and performed before each of the analyses. • CORE also excluded all missing data observations before the specific analyses that were conducted for hundreds of different conditions. I Appendix B: Exploration of Effects of Campus Level TFA Saturation on Student Achievement Exploring Saturation CORE also explored the potential influence of “saturation” of TFA in a given context, hypothesizing that there could be setting-level effects of clustering TFA corps members and/or alumni that indirectly influence student outcomes. These analyses are preliminary. More in-depth analysis is Summary: CORE calculated a TFA saturation planned for future reporting in early 2019. Methodology The orienting question for saturation was: are observed impacts greater when a student is in a campus environment with more TFA corps members or alumni? These analyses used the same data cleaning and merging strategies described for the impact analyses, and using the same exclusion criteria described above. In addition to these criteria, CORE selected the schools that had at least one TFA-affiliated teacher for the saturation analyses; all students with a zero TFA-affiliated teacher percentage for their school were excluded from the datasets. rate for campuses and used linear regression analyses to determine the association between having a greater percentage of TFA-affiliated teachers and students STAAR achievement. Results show that across all analyses, 53% of the time, there was a positive and significant association between a greater TFA saturation on a campus and student academic achievement on STAAR. Overall, while the association between campus-level saturation and student outcomes was positive and significant, the findings are not practically meaningful. In other words, the positive effect is small. This effect was most pronounced in high school where the vast majority of analyses revealed a positive and significant association between saturation of TFA and student STAAR scores. CORE derived the school-level TFA-affiliated teacher saturation measure for each student by using the head counts of TFA and non-TFA-affiliated teachers in the students school matched to that student in a given school year. The “saturation” measure is the percent of all teachers on a campus that are TFA-affiliated. For example, if half of the teachers on a campus are TFA corps members or alumni, the saturation score is 0.5. All students that had a zero percentage of TFA-affiliated teacher saturation were excluded from the entire analyses. Similar to previous analyses, CORE created 158 different conditions depending on various demographic variables, areas, and school years. CORE used the linear regression approach to model the effect of TFA-affiliated teacher saturation within a school on the academic achievement of students. The continuous scaled scores for each of the STAAR subjects was used as the outcome measure. The predictor of interest was the derived TFA-affiliated teacher saturation at a student’s school in a given year (only including students with saturations greater than zero). All covariates used in the PSW procedure for balancing were also used during the saturation analyses as controlling variables. Thus, the effect of the saturation on student outcomes was examined by controlling all available demographic variables of students, schools, and even districts. CORE also conducted some disaggregation analyses for each category of the demographic variables in order to take a deeper look at the saturation effect. For example, separate analyses were conducted to test the effect of saturation on individual student race groups to determine if the saturation had a differentiated effect on these unique groups of students. Regression analyses provided the statistical significance of the saturation effect, and two types of effect sizes that represents the magnitude of that effect. The standardized Beta coefficient for the saturation measure can be considered a general type of effect size in the scale of the outcome standard deviation. A minimum value of 0.2 is considered important in terms of Beta coefficient, and in our case, this value would indicate that the score of the students will increase 0.2 standard deviations, by an increase of one standard deviation of TFA-affiliated teacher percentage. CORE also provided the Cohen’s f values as a second type of effect size measure. This value is calculated II by using the amount of the variance explained by the variable of interest itself, which is the TFA-affiliated teacher saturation in our case. The thresholds for small, medium, and large Cohen’s f values are 0.02, 0.15, and 0.35, respectively (Cohen, 1988; Selya et al., 2012). As noted earlier, the interpretation of the effect sizes should be done in line with the research question and context. Thus, even a small effect size can be considered important, if the Beta coefficient is positive and statistically significant. Saturation Results This question examined whether student STAAR scores are predicted to be higher if a student attends a school with a relatively higher saturation of TFA-affiliated teachers, including the alumni and corps members. Similar to previous analyses, CORE created 158 different conditions depending on various demographic variables, areas, and school years. CORE used the linear regression approach to model the effect of TFA-affiliated teacher saturation within a school on the academic achievement of students. Table A.1 provides descriptive statistics about the saturation variable. On average, across all 6 years in the study, 7.3% of the teachers in a student’s elementary school (grades 35) were TFA-affiliated. The average saturation for middle schools and high schools is 8.3% and 3.6%, respectively. Schools with zero TFA-affiliated teachers were removed prior to the analyses. Table A.1. Descriptive statistics of campus TFA saturation at the student-level By School Level: Elementary Grades Middle Grades High School Grades By Area: Region A Region B Region C Region D Region E Min 0.4% 0.5% 0.3% Max 56.8% 100.0% 69.6% Mean 7.3% 8.3% 3.6% Sd 7.5 9.9 5.7 0.3% 0.3% 0.3% 0.4% 0.4% 53.8% 69.6% 100.0% 58.3% 65.0% 3.4% 6.5% 5.3% 4.5% 5.7% 7.1 8.1 8.5 6.7 6.0 Figure A.1 describes the overall distribution of the findings across the 158 analyses. The result of each of the 158 analyses was categorized into one of four groups indicating both (1) weather a higher saturation of TFA-affiliated teachers on a campus predicted higher student achievement on that campus, and (2) the statistical significance of the finding. The four categories are described in Table A.2. In 53% of all saturation analyses conducted, a significant positive relationship between TFA saturation on a campus and student performance on that campus was observed. This means that about half of the time, having a higher percentage of TFA-affiliated teachers on the campus was associated with better academic outcomes for students and about half the time it was not. Table A.2. Categories of direction and significance of effect of campus-level TFA-affiliated teacher saturation on student performance Finding category 4 3 2 1 As saturation of TFA-affiliated teachers increases (Corps Members or Alumni) on a campus, student performance on that campus also increases yes yes no Result is statistically significant yes no no no yes III Figure A.1. Distribution of TFA saturation analyses into direction & significance findings categories; n=158 total analyses 4 = positive & significant 3 = positive but not significant 2 = negative but not significant 53% 13% 1=negative & significant 11% 23% The pie charts below (Figure A.2) describe how the subject-specific saturation findings are distributed. In other words, these pie charts give a detailed view of how increased saturation of TFA-affiliated teachers on single campus is predicted to increase the performance of students on that campus in specific content areas. As seen in the pie charts below, saturation of TFA-affiliated teachers on a campus was consistently associated with greater student achievement on the End of Course exams. In other words, for high schools specifically, more TFA-affiliated teachers is positively and significantly associated with greater student performance in Algebra 1, Biology, English 1 and English 2 in 85% of cases, on average. In elementary & middle schools, higher TFA saturation most often had a positive effect on Reading. Figure A.2. Distribution of TFA saturation analyses into direction & significance findings categories by content area Elementary and Middle School Core Subjects: Reading (n=18 analyses) Math (n=18 analyses) 11 % 6% 22 % 6% 28% 11 % 61 % 11% Science (n=18 analyses) 72 % 28 % Social Studies (n=18 analyses) 22 % 22% 39 % 17% 22% 22% High School End-of-Course Exam Subjects: Algebra 1 (n=18 analyses) Biology (n=18 analyses) 6% 22 % English 1 (n=16 analyses) 13% English 2 (n=16 analyses) 6% 13% 17% 6% 72 % 94% US History (n=18 analyses) 88 % 88 % 61 % 17 % CORE also explored the Cohen’s f values as a second type of effect size measure. This value is calculated by using the amount of the variance explained by the variable of interest itself, which is the TFA-affiliated teacher saturation in our case. The thresholds for small, medium, and large Cohen’s f values are 0.02, 0.15, and 0.35, respectively (Cohen, 1988; Selya et al., 2012). As noted earlier, the interpretation of the effect sizes should be done in line with the research question and context. Thus, even a small effect size can be considered important, if the Beta coefficient is positive and statistically significant. The effect size was very low for all but one of the 158 analyses, near zero (range: 0.000 – IV 0.006). In other words, while saturation of TFA may have a positive or negative effect on student performance, the magnitude of that effect may be interpreted as having little to no practical significance. Table A.3. Differentiated effect of TFA campus-level saturation on student achievement for demographic subgroups Charter schools and For charter schools, findings are significantly positive 67% of the time, traditional ISD schools compared to 44% for traditional ISD campuses. Student race For black students, findings are significantly positive 67% of the time, compared to 44% for White and Hispanic students. Economically For economically disadvantaged students, findings are significantly positive disadvantaged 56% of the time, compared to 44% for non-economically disadvantaged students. Limited English proficient For limited English proficient students, findings are significantly positive 78% of the time, compared to 56% for non-limited English proficient students Campus performance For campuses that meet state standards, findings are significantly positive 56% of the time, compared to 33% for campuses that do not meet state standards. V Appendix C: Results of ANOVA Meta-Analysis of all 699 Impact Analyses Tests of Between-Subjects Effects Type III Sum Source of Squares df Mean Square F Sig. TFA affiliation 4.858 2 2.429 50.128 .000 Geo Area 2.065 5 .413 8.523 .000 Subject 2.793 7 .399 8.234 .000 Geo Area * Grade 4.456 27 .165 3.406 .000 Geo Area * Subject 4.177 28 .149 3.079 .000 Geo Area * Grade * Subject 3.451 27 .128 2.637 .000 Grade .701 7 .100 2.066 .052 Grade * TFA affiliation * Subject .984 14 .070 1.450 .140 Geo Area * TFA affiliation .551 8 .069 1.421 .194 Grade * TFA affiliation .857 14 .061 1.264 .239 Geo Area * Grade * TFA affiliation 2.627 44 .060 1.232 .185 Geo Area * TFA affiliation * Subject 2.144 38 .056 1.164 .263 TFA affiliation * Subject .692 14 .049 1.020 .438 Grade * Subject .270 7 .039 .795 .593 Geo Area * Grade * TFA affiliation * Subject .692 40 .017 .357 1.000 Error 6.154 127 .048 Total 57.681 439 Corrected Total 52.228 438 This table presents the findings of the ANOVA conducted on 699 different impact analyses results. The dependent variable of the model was the estimated effect size of each impact analyses. Independent variables were the factors describing each analysis, such as TFA-affiliation (brand new corps members/all corps members/alumni), geographic area, and STAAR subject. Highlighted main and interaction effects are the factors that significantly predict the effect size of the analyses. Thus, it can be interpreted that the effect sizes from various analyses differed depending on: • • • • • TFA-affiliation Geographic area STAAR Subject (e.g., Reading, Math, Algebra 1) Two way interaction of the area with grade and subject Three way interaction of area, grade, and subject VI Appendix D: Comparison Analysis Heat Map This heat map describes the direction and significance of the comparison analyses findings by subject, grade level, TFA affiliation, and geographic region. Blue spaces indicate a significant finding favoring TFA-affiliated teachers, and black spaces indicate a significant finding favoring the non-TFA-affiliated teachers 6. Region A Region B Region C Region D Region E Heatmap legend: ALM=alumni, CM-A=all corps members; CM-1=first year corps members, A1=Algebra 1, BI=Biology, E1=English 1, E2=English 2, M=Math, R=Reading, SS=Social Studies, US=US History 6 VII Appendix E: Summary of Comparison Analysis Findings by Demographic Subgroups The table below describes which of the comparison analyses that positively and statistically significantly favored TFAaffiliated teachers for specific demographic subgroups, as organized by content area. To the right of the dotted vertical line, each space represents one unique analysis (e.g., Algebra 1 for economically disadvantaged students taught by TFA alumni). Spaces marked n/a did not have a valid analyses for that specific combination of factors, most often due to small sample size or exact probability prediction (1/0), which is considered invalid. Yes EcoDis No Blk Race/ Ethnicity Hisp Wht Charter School Type ISD Yes LEP Status No School Status Met Stndrd Not Met Stndrd Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Alumni 1st yr CMs All CMs Proportion of Content Areas Significantly Positive for TFA Group 88% 22% 33% 83% 13% 0% 88% 38% 0% 88% 11% 44% 50% 33% 0% 56% 33% 22% 100% 75% 33% 88% 22% 22% 88% 50% 56% 88% 22% 11% 100% 50% 13% Each space within the content-specific columns represents one analysis that was conducted for that subject under those specific conditions (as indicated by the row indicators to the left of the table). Yes = analysis significantly favored the TFA group. Alg1 yes yes No yes No No yes yes No yes No No n/a n/a No yes No No yes yes No yes No No yes yes yes yes No No yes yes No Bio yes No yes n/a No No yes yes No yes No yes n/a No No No No No yes yes yes yes yes No yes No yes yes No No yes yes yes VIII Eng1 yes No No No No No No No No yes No No No n/a n/a yes yes yes yes No No yes No No yes yes yes yes No No yes No No Eng2 yes No No n/a No No yes No No yes No yes n/a n/a n/a yes No No yes yes yes yes No No yes yes yes yes No No yes No No U.S. Hist n/a No yes n/a n/a n/a n/a n/a n/a n/a No yes n/a n/a n/a No yes yes n/a n/a No n/a yes yes n/a n/a yes n/a yes yes n/a n/a n/a Math yes No No yes No No yes No No yes No No yes No No yes No No yes yes No yes No No yes No No yes No No yes yes No Rdg yes No No yes No No yes No No yes No No n/a yes No No No No yes yes No yes No No yes No No yes No No yes No No Sci yes yes yes yes yes No yes yes No yes yes yes n/a n/a n/a yes yes No yes yes yes yes No yes yes yes No yes yes No yes yes No Soc Stud no No No yes No No yes No No No No No n/a n/a n/a No No No yes No No No No No No No No No No No yes No No