The Impact of Math Circle Curriculum on Student Learning: A Pilot Study

pi_name

Oleg Gleizer

pi_email

prof1140g@math.ucla.edu

pi_phone

(310) 825-4462

pi_department

Department of Mathematics

pi_title

Professor

other_key_personnel

Jia Wang, Ph.D.
ADJUNCT PROFESSOR
ASSISTANT DIRECTOR
NATIONAL CENTER FOR RESEARCH ON EVALUATION,
STANDARDS, AND STUDENT TESTING
300 CHARLES E. YOUNG DR. NORTH, GSEIS BLDG. RM. 307
LOS ANGELES, CA 90095-1522
PH: (310) 267-4476 | WWW.CRESST.ORG

abstract

We propose to conduct a pilot study aimed at investigating the impact of teaching UCLA Olga Radko Endowed Math Circle (ORMC) math curriculum on elementary school students’ math performance at the UCLA Lab School in 2023-2024. The ORMC curriculum will be offered to students as a weekly after-school program at no cost to students. We will randomly assign grades 3-6 students to be ORMC participants or non-ORMC students. The assignment is valid for all three ORMC sessions in the 2023-24 school year. Both groups of students are expected to take two pre-tests in math at the beginning of the school year and the same two post-tests at the end of the school year to evaluate the effect of the intervention. Both tests are not related to the ORMC curriculum. We hypothesize that studying the ORMC curriculum makes participating students score/perform at a higher level than the control students in this randomized control trial study.

project_summary

We propose to conduct a pilot study investigating the impact of ORMC math curriculum on elementary school students’ performance in math in 2023-2024, at UCLA Lab School. The main study question is: What is the impact of Math Circle intervention on elementary school students’ math achievement? The ORMC curriculum we intend to use was developed by the late Prof. Olga Radko, the founding director of the Circle, and by Prof. Oleg Gleizer, the current ORMC director. The evaluation will compare the ORMC students to non-ORMC students based on random assignment. The student demographic information collected by the Lab School will be used as covariates in the analysis in addition to the to-be-collected pre-test math scores. The post-test math scores will be used as the study outcome measure.

The proposed target student population is those in grades 3-6. As of July 2023, the Lab School has 122 students in grades 3-4 and 79 students in grades 5-6. The plan is to offer two ORMC classes, with one class for students in grades 3-4, and one class for students in grades 5-6. The enrollment cap is 25 students per class. Each class is to be taught by two instructors. Prof. Gleizer will recruit and train UCLA undergraduate students who will be serving as study instructors. The math curriculum will be offered to treatment students as a weekly extra-curriculum program at no cost to students. The program consists of three sessions of 8-week instruction during the school year. The instruction is scheduled to be 50 minutes. In addition, students will be assigned a moderate amount of homework.

All students in grades 3-6 are expected to take two math pre-tests at the beginning of the school year and the same two math post-tests at the end of the school year. The combined time allocated for the two pre-tests is 60 minutes and the same for the two post-tests per the UCLA Lab School.

The first test, Renaissance Star Math, is part of the Renaissance Star assessments. More than 1/3 of the schools nationwide use Renaissance Star assessments and more than 58 million assessments are administered online each school year. Renaissance Star Math will take students 20 mins to complete.

The other test will be an 8-item test for Grades 3-4 students or a 9-item math test for Grades 5-6 students, selected from the 2010 math competition tests called Math Kangaroo. The regular Math Kangaroo test has 25 items and we made a shortened version so students can complete the test in 30 minutes.

The end-of-quarter student survey will be given to math circle students as the assignment that will be due when students come to week 8 instruction. The participating students will be expected to complete the survey in the privacy of their own homes.

The staff at the Lab School plan to include the ORMC program as one of the after-school extra-curriculum programs for students. A program flyer is prepared and attached. The tentative days of the week are either Tuesday and/or Thursday afternoons to avoid time overlap with other after-school programs at the Lab School. Parents are invited to register their students for the program/study.

Prof. Jia Wang, an expert in social science research methodology and the leader of the evaluation work at CRESST as the Assistant Director of Program Evaluation, will be conducting an independent evaluation of the program’s impact in a randomized control trial study.

goals

We are proposing to conduct a pilot study investigating the impact of the ORMC on students’ performance in math at the UCLA Lab School in 2023-2024. When the intervention is well implemented, we expect the study to show that students are performing at a higher level in the post-tests and we expect the study to meet the evidence of promise standard minimally as established by the U.S. Department of Education.

benefits_of_research

It is to improve the math educational outcomes across all students in elementary grades and to provide useful information to the public and to policymakers about this effective strategy.

dissemination/publications

A technical report describing the study will be prepared and disseminated. We will present the study findings at educational conferences and prepare a journal article for publication.

numer_of_subjects

200

selection_criteria

There are no selection criteria besides requiring the students to be in grades 3-6 in the 2023-2024 school year.

methods

This is a randomized control trial study. Our goal is to offer two ORMC classes, with one for students in grades 3-4, and one for students in grades 5-6. The enrollment cap is set at 25 students per class.

The Math Circle program description will be shared with the Lab School staff and included in the enrichment offerings at the Lab School website along with other offerings. Interested parents can register their students in the ORMC program by clicking the link or scanning the bar code.

The registration will ask the parents to provide parent/guardian email addresses, student names, and student grade levels. The information will be collected using a UCLA Qualtrics account.

Meanwhile, the CRESST team will request the Lab School classroom rosters and use them to randomly assign students to either treatment (receiving ORMC curriculum) or control (no ORMC curriculum) status by grade level.

Parents will be notified of their student's study status after the registration close. The parents of students who are randomly selected for ORMC participation will also receive the parent consent form and the solicitation for a donation flyer. These parents have about one week to confirm their student's participation and to sign the consent form.

Student Math Assessment: Star math assessment and a shortened version of the 2010 Math Kangaroo will be used. This is confirmed with Prof. Gleizer and Ms. Smith before finalizing the decision.
• Star math assessment: CRESST staff will request and receive the classroom roster from the Lab School at the beginning of September 2023 to set up student accounts for test administration. CRESST staff will generate and share the student login and password with each applicable teacher in grades 3-6 for them to share with their students. During the pre-designated time, the teachers will have their students log into the Star math assessment account and complete the assessment. It is projected to take 20 minutes.
• Math Kangaroo assessment: CRESST staff will make copies of the shortened 2010 version of the Math Kangaroo assessment and distribute them to applicable teachers in grades 3-6. It is projected to take 30 minutes.
• For pre-tests, both assessments are scheduled to be administered in the week of 9/25/23 on two separate days of the week.
• For post-tests, we plan to have both assessments administered in mid-May 2024.

Student Survey. Prof. Wang will develop a short student survey asking about students’ experiences and opinions of the ORMC curriculum at the end of each 8-week session. The end-of-quarter student survey will be given to math circle students as the assignment that will be due when students come to week 8 instruction. The participating students will be expected to complete the survey in the privacy of their own homes.

Analysis Approach. To examine the ORMC impact on measurable student outcomes, we will use a regression-based approach with bias adjustment, which performed well in a simulation study as detailed in Huber et al. (2012).

instruments

Warning: Array to string conversion in /opt/data/www/connect/wp-content/themes/ucla-connect-test/templates/content-single-project.php on line 29
Array

instruments_other

NA

instrument_explanations

Assessment in mathematics is needed to measure student performance and improvement in their math performance at the beginning and end of the participation.

The student survey is to get students’ perspectives on their learning experience and their perception of the program.

justification_of_methods

The proposed pilot study will be a randomized control trial study, comparing the ORMC students to the non-ORMC students (control students) at the same school. Prof. Wang and her team at CRESST have been perfecting the use of rigorous methodological approaches to the analysis of student outcome data in quasi-experimental design (QED) and RCT studies. The QED approach has been applied to about 50 federally funded grants including Investing in Innovation (i3) Program, Magnet Schools Assistance Program (MSAP), and Teacher Quality Partnership (TQP) Program. The RCT approach is now being applied in an Education Innovation and Research (EIR) Program and an American History and Civics Program.

separate_informed_consent

Yes for the parents of ORMC students to consent to the study

risk_minimization

There are no potential risks/discomforts. It is common for students to participate in academically oriented after-school programs at schools. There is no reasonable concern that the short surveys and math assessments could pose any risks for students.

Students may withdraw anytime during the study.

deception_debriefing

NA

confidentiality_data_storage

The study will collect only the minimum identity information needed. Secure network server will be used to store the collected data. As for the data we collect, we remove identifiers from analytic files as soon as they are no longer needed. The identifiers are stored in a physically secure location separate from the data files and only the PIs will have access to the code key. Identifiers are not stored on laptops, PDAs, flash drives, or other portable devices.

All study personnel are required to complete UCLA IRB training and have a valid certificate to conduct research.

other_notes

NA

relationship_prior_contact

Prof. Gleizer and Prof. Wang had several meetings with the Lab School staff including Prof. Megan Franke, Ms. Sandra Smith and Dr. Lisa Baggio.

teachers_staff_consent

Lab School Administration has approved the offering of Math Circles for La School extended day program

ucla_lab_school_personnel_involved

Prof. Gleizer and Prof. Wang had several meetings with the Lab School staff including Prof. Me-gan Franke, Ms. Sandra Smith and Dr. Lisa Baggio.

academic_topic

Math

information_from_ucla_lab_school_database

Classroom rosters for random assignment

Student-level data if applicable and if available

Student ID
grade level
gender
race/ethnicity
free/reduced lunch status
English Proficiency status (English Only, English Learner, etc.)
special education status
gifted status
years in Lab School
total days enrolled for 2023-24
total days attended for 2023-24
total days absent for 2023-24
Math course title enrolled for Fall 2023
Math course GPA for Fall 2023
Math course title enrolled for Spring 2024
Math course GPA for Spring 2024

special_requirements_at_ucla_lab_school

The after-school ORMC program will be offered in the Lab School classroom as part of the enrichment program offerings on the Lab School campus. We rely on Lab School Staff to include our program offering in the general offering website. Additionally, we rely on Lab School Staff to share the classroom rosters for the random assignment, share the typical student data the School collects on students, allocate time for testing, and help with getting iPads to students for the testing.

estimated_start_date

20230906

estimated_end_date

20241220

irb

IRB#23-001062

irb_approval

Accepted pending modifications

attachments

Warning: Array to string conversion in /opt/data/www/connect/wp-content/themes/ucla-connect-test/templates/content-single-project.php on line 29
Array