For the third consecutive year, the Harvard PEAR Institute has provided year-end data assessing the performance of 11 youth-serving informal STEM programs—consisting of student-level measures covering STEM-related attitudes and SEL competencies, program quality, and facilitator feedback—with comparisons to national norms and prior year results.
Highlights:
Over 2,700 students across 11 programs submitted end-of-year retrospective surveys (92% in grades 4-8, 82% under-represented minorities, 40% whose primary language is not English, and 42% who had no prior experience in out-of-school STEM activities).
Across all 10 measures covering STEM-related attitudes and SEL competencies, Tulsa students had, on average, higher gains than nationally-comparative norms.
Regarding STEM attitudes:
88% of Tulsa students reported positive gains in STEM interest, 89% reported gains in STEM career interest, 80% reported gains in STEM career knowledge (whether students know what steps to take if they wish to pursue a STEM career), and 67% reported gains in STEM identity.
Regarding SEL competencies:
85% reported gains in critical thinking (Ex: “I like to think of different ways to solve a problem”), 83% reported gains in perseverance (Ex: “I work hard to achieve goals even if things get in the way”), and 79% reported gains in adult & peer relationships.
Across all 10 measures, Tulsa girls reported higher gains than Tulsa boys and the national comparison groups of girls and boys with significantly higher gains in STEM enjoyment, STEM career interest, and STEM career knowledge.
Looking at year-over-year trends from 2016-2018, Tulsa STEM programs have improved significantly in terms of students reporting positive gains in STEM career interest (up 30 ppt.), STEM career knowledge (up 23 ppt.), and student motivation to seek out STEM activities (up 32 ppt.).
Program quality:
Over the past two years, the STEM Alliance has observed 36 programs using PEAR’s Dimensions of Success (DoS) program quality tool—assessing programs on 12 standards (covering features of the learning environment, activity engagement, STEM knowledge & practices, and relevance)—revealing that Tulsa is outperforming national norms on 5 of the 12 measures, and since 2016 has met or exceeded the bar for quality (3.0 rating or above) on 7 of the dimensions.
Next steps:
The STEM Alliance has developed an action plan for the next year focused on program quality improvement with the objective of “enabling STEM programming organizations to build an internal culture of continuous improvement so that front-line youth-serving staff have the structures, tools, and support they need to strengthen program delivery.”
Significant pieces of work include:
- Reflection on data/trends
- Identification of high-leverage areas for improvement, supporting utilization of program planning tools and STEM learning resources
- Ensuring continuous quality improvement practices among partner organizations via DoS observations and feedback sessions.
To kick off the year, TRSA and PEAR will facilitate a Continuous Quality Improvement Community of Practice convening on September 19th for Tulsa area STEM organizations to provide a re-orientation to resources, get buy-in on CQI practices, and build customized staff training plans that each organization can implement over the school year.
Overall, this data and support from PEAR has provided the local STEM education community with a common language for program quality, a common understanding of student-level intended outcomes tied to program participation, and access to data/trends to inform the STEM Alliance’s efforts to support organizations in building cultures of continuous quality improvement.
