Evaluation Context: Capturing a Culture of Evaluation in a University Laboratory School Setting

pi_name

Christine Abagat Liboon

pi_email

clibo001@g.ucla.edu

pi_phone

(619) 948-6927

pi_department

Graduate School of Education and Information Studies

pi_title

Graduate Student Researcher

ucla_faculty_sponsor

Dr. Christina (Tina) Christie

other_key_personnel

Dr. Nicole Mancevice

abstract

project_summary

The purpose of this study is to see how a culture of evaluation is created within a laboratory school setting. Evaluation theorist Marvin Alkin shared how important context is to evaluation using the following analogy: “in real estate, it’s location, location, location The evaluation counterpart is context, context, context.” Drs. Anne Vo and Christina Christie mentioned that research in evaluation and the practice of evaluation has widely benefited from the knowledge gained from studies that examine when evaluation is used (evaluation use) and when participants involved in the evaluation process are engaged in a specific method driven by theory (stakeholder engagement, prescriptive theory) but evaluation context itself “is a topic that remains under-investigated.”

Context, in this study, is defined as the interactions between the administrators, the development director, in partnership with the university’s education department research director and assistant researchers. The results that come from this study have broader implications on how a culture of evaluation can be established when participants who are involved in the process are all engaged (stakeholders, researchers, evaluators) - in the case of this study, a university laboratory school setting. Context is a crucial part of evaluation research and for this reason, Drs. Vo and Christie stress it’s necessary to develop a systematic way to study it. However, an existing systematic design to capture evaluation context has yet to be explored.

This study asks the following research questions: (1) what is the context in which a culture of evaluation is established at a laboratory school? And, (2) how does the school and research team use the results from the alumni survey for the alumni network?

In this proposal, I will conduct a qualitative research study and document the following processes through the following data collection methods:
1) Meetings observations (via Zoom) in which the survey is developed or discussed by Lab School and study researchers. There will be no in-person interactions at this time due to COVID19. These meetings will be audio and video recorded and transcribed via Zoom software using pseudonyms/codes for participants. Participants who will be included in these meeting activities may include the school’s administrators, the development and alumni relations director and team, the research team, and any party involved in the survey development, implementation, and follow-up process.
2) If a participant declines during the individual private oral consent process, the researcher/principal investigator will provide the options to omit information and participation: audio recording via Zoom only, neither video nor audio recording - notes only. The researcher/principal investigator will remove information the participant wishes to leave out.
3) Memos, journals, or emails originating from the principal investigator.
4) I will use observations to conduct a qualitative case study around the process of the development, implementation and follow-up of an alumni survey developed for the Lab School’s Alumni and Development team to document the context of evaluation research and use data collected from documents, artifacts, and others described to support this process. The survey itself targets student alumni who have exited the K-12 system (18+ years old) to learn about their experiences and perceptions of the UCLA Lab School, while developing a larger alumni network for the school. Data results from the survey will not be used in this study – the process around the development, implementation, and follow-up will be the focus of this study.
5) The principal investigator’s personal notes of meetings with the Lab School Alumni and Development team which took place prior to the start of this project from May 2019 to August 2020 will be requested to be used in this study.
6) Collection of publicly available information on the Lab School. This may include journal articles, dissertations, and information made publicly available on the internet such as through available media sources (website, blogs, journals, news articles).

goals

The results that come from this study have broader implications on how a culture of evaluation can be established when participants who are involved in the process are all engaged (stakeholders, researchers, evaluators) - in the case of this study, a university laboratory school setting. An existing systematic design to capture evaluation context has yet to be explored.

This study will be presented at conferences (AERA, AEA) and a written document (brief, article, white paper, etc.) will be produced and submitted to a relevant source (journal, school website, university institute website, etc.).

benefits_of_research

Context is a crucial part of evaluation research. There is limited information on an existing systematic design to capture evaluation context. The results from this study will provide useful information about the way in which a culture of evaluation is developed and created in a university laboratory school setting. In addition, this case study will strengthen existing knowledge, such as evaluation literature in research, on ways to provide a systematic approach to capture evaluation context.

Expanded knowledge that result from this study will help to inform the Lab School's Alumni Relations and Development Teams as they gain knowledge through the survey development, implementation, and follow-up process of alumni's (ages 18+ years old) impressions on their Lab School experience as well as how to better understand the use of the results from the survey throughout this process.

dissemination/publications

The study seeks to understand the context of evaluation and strengthen the literature on ways to provide a systematic approach to capture evaluation context. The study products that come from this study (papers and presentations) will be submitted to several educational conferences, including AERA (American Educational Research Association), AEA (American Evaluation Association) and CREA (Culturally Relevant Evaluation and Assessment). Findings will be shared with the Lab School’s CONNECT to Research team, school administrators, and development director to inform current practices.

numer_of_subjects

selection_criteria

Participants who will be included in observational activities may include the school’s assistant principal, the development and alumni relations director and team, the research team, and any party involved in the survey development and implementation process. An invitation to participate will take place in two ways, either by email or during a Zoom meeting which will take place in August 2020 and September 2020. See recruitment scripts (direct recruitment and recruitment by message/email).

methods

In this proposal, I will be documenting the following processes through the following data collection methods:

1) Meetings observations (via Zoom) in which the survey is developed or discussed by Lab School and study researchers. Meetings for observation will take place in a private Zoom room using a unique link. There will be no in-person interactions at this time due to COVID19. These meetings will be audio and video recorded and transcribed via Zoom software using pseudonyms/codes for participants. Participants who will be included in these meeting activities may include the school’s administrators, the development and alumni relations director and team, the research team, and any party involved in the survey development, implementation, and follow-up process.
2) If a participant declines during the individual private oral consent process, the researcher/principal investigator will provide the options to omit information and participation: audio recording via Zoom only, neither video nor audio recording - notes only. The researcher/principal investigator will remove information the participant wishes to leave out.
3) Memos, journals, or emails originating from my own correspondence as the lead researcher on this study with participants.
4) I will use observations to conduct a qualitative case study around the process of the development, implementation and follow-up of an alumni survey developed for the Lab School’s Alumni and Development team to document the context of evaluation research and use data collected from documents, artifacts, and others described to support this process. The survey itself targets student alumni who have exited the K-12 system (18+ years old) to learn about their experiences and perceptions of the UCLA Lab School, while developing a larger alumni network for the school. Data results from the survey will not be used in this study – the process around the development, implementation, and follow-up will be the focus of this study.
5) I will use my personal notes of meetings with the Lab School Alumni and Development team which took place prior to the start of this project from May 2019 to August 2020 and will request from participants to be used in this study.
6) Collection of publicly available information on the Lab School. This may include journal articles, dissertations, and information made publicly available on the internet such as through available media sources (website, blogs, journals, news articles).

For the study’s purposes around the development, implementation, and analysis of the survey, any information collected on the school staff, community members and alumni participants, the principal investigator and the research team will create codes (i.e., pseudonyms). Only myself as the principal investigator and my research team (non-Lab School staff) will have access to the codes. Identifying information will be removed and destroyed at the close of the study.

Interviews will be proposed in the future of this study with individuals involved in the development, implementation, and follow-up of the survey – these interviews will take place when regular Lab School research activities resume during Winter/Spring quarter 2021. I will complete an amendment (with the IRB and the Connect to Research Application) with the interview protocol when given permission and regular research activities resume.

instruments

Warning: Array to string conversion in /opt/data/www/connect/wp-content/themes/ucla-connect-test/templates/content-single-project.php on line 29
Array

instruments_other

N/A

instrument_explanations

Observation Protocol: I will be using data an observation protocol to guide the notetaking process. The following protocol above will be used to:
• Jot down any notes during onsite observations, meetings, or shadowing of any staff members (online Zoom meetings due to COVID19)
• Observe how assistant principal, coordinators, and research team prepare, implement, and deliver feedback in the Lab School context.
• Capture any observations on context and interaction processes during the length of survey preparation and implementation.
• No personal nor identifying information will be captured in the process of jotting. Researcher will use pseudonyms in the process of notetaking to protect the confidentiality of any subjects.

Survey: Though the results from the survey will not be used in this study, I include it here for reference and context. The survey targets student alumni who have exited the K-12 system (18+ years old) to learn about their experiences and perceptions of the UCLA Lab School, while developing a larger alumni network for the school. The survey is an integral tool for the Alumni and Development team to understand previous alumni’s thoughts. It will be the primary instrument in how conversations are shaped for the researcher to understand the context in which the survey is used by its stakeholders. Results generated from the survey and its analysis will not be directly used in this study – rather the conversations broadly about and around its development, implementation, and follow-up.

justification_of_methods

Though the results from the survey will not be used in this study, it will be the central tool used within the process to understand the interactions around the context of evaluation during its development, implementation, and follow-up. I would like to document the conversations around the stages of the survey’s development, implementation, and follow-up. Gathering notes around this process will help to inform how decisions are made regarding the creation of the survey and the use of survey responses. For this reason, I would like to record meetings. This will primarily take place around the Alumni Relations and Development team, myself as the researcher and other CONNECT Research / UCLA GESIS faculty/staff we will collaborate alongside. Moreover, document collection and analysis will further provide contextual information on the school’s history and context for the background of this study.

separate_informed_consent

See attached oral consent form which will be delivered individually to each participant online over Zoom.

risk_minimization

Participants will also be asked for their oral consent, privately and individually in a Zoom room over the internet prior to participating in the study. Meetings will take place in a private Zoom room using a unique link due to COVID19. Participating members include those involved in the development, implementation and follow-up of the survey. There will be no in-person interactions at this time. If that changes, interactions will take place in a private room provided by the UCLA Lab School. Upon transcribing any notes, the researcher will use pseudonyms to protect the privacy of the research team and use non-identifying information.

deception_debriefing

N/A

confidentiality_data_storage

The contact information provided, if any, (phone number and/or email) collected during recruitment will be deleted from study records at the conclusion of data collection.

For the study’s purposes around the development, implementation, and analysis of the survey, any information collected on the school staff, community members, and alumni participants, the principal investigator and the research team will create codes (i.e., pseudonyms). Only the principal investigator and research team (non-Lab School staff) will have access to the codes. Identifying information will be removed and destroyed at the close of the study.

A secure network server will be used to store data. Any hardcopy data will be locked in a secure place (locked file cabinet). Digital files will be stored on UCLA's BOX cloud storage software, a university-approved, secure method to store and share documents online. All data files will be stripped of personal identifiers and/or the key to the code destroyed.

The contact information provided, if any, (phone number and/or email) collected during recruitment will be deleted from study records at the conclusion of data collection.

For the study’s purposes around the development, implementation, and follow-up of the survey, any information collected on participants, the principal investigator and the research team will create codes (i.e., pseudonyms). Only the principal investigator and research team will have access to the codes. Identifying information will be removed and destroyed at the close of the study.

other_notes

In the future, I would like to conduct interviews in this study with individuals involved in the development and implementation of the survey will take place when regular Lab School research activities resume during Winter/Spring quarter 2021. I will complete an amendment (with IRB and the Connect to Research) with the protocol when given permission and regular research activities at the Lab School resume.

relationship_prior_contact

I (the researcher) am a graduate student at the UCLA Graduate School of Education and Information Studies and have been introduced to Lab School staff through my advisor-Dr. Christina (Tina) Christie-and instructor -Dr. Nicole Mancevice- for this project.

teachers_staff_consent

I will be seeking staff consent for meetings prior to being recorded during meetings. Verbal consent at the time of the meeting (see oral consent script). For future interviews, written consent forms and their signatures will be distributed and collected digitally.

ucla_lab_school_personnel_involved

Assistant Principal, the Alumni Relations and Development Team, CONNECT Research Team

academic_topic

Evaluation Research - Evaluation Context - Development at a Laboratory School

information_from_ucla_lab_school_database

N/A

special_requirements_at_ucla_lab_school

N/A

estimated_start_date

20200615

estimated_end_date

20210615

irb

IRB#20-001436

irb_approval

apoproved 9/30/20

attachments

Warning: Array to string conversion in /opt/data/www/connect/wp-content/themes/ucla-connect-test/templates/content-single-project.php on line 29
Array