pi_name
Christine Abagat Liboonpi_email
clibo001@g.ucla.edupi_phone
(619) 948-6927pi_department
Graduate School of Education and Information Studiespi_title
Graduate Student Researcherpi_title_other
Graduate Studentucla_faculty_sponsor
Dr. Christina (Tina) Christieother_key_personnel
Dr. Nicole Manceviceabstract
The purpose of this study is to see how a culture of evaluation is created within a laboratory school setting. Evaluation theorist Marvin Alkin shared how important context is to evaluation using the following analogy: “in real estate, it’s location, location, location. The evaluation counterpart is context, context, context.” Drs. Anne Vo and Christina Christie mentioned that research in evaluation and the practice of evaluation has widely benefited from the knowledge gained from studies that examine when evaluation is used (evaluation use) and when participants involved in the evaluation process are engaged in a specific method driven by theory (stakeholder engagement, prescriptive theory) but evaluation context itself “is a topic that remains under-investigated.” This study asks the following research questions: (1) what is the context in which a culture of evaluation is established at a laboratory school? And, (2) how does the school and research team use the results from the alumni survey for the alumni network? Data collected includes: observations from meetings around the development, implementation, and follow-up process from an alumni survey; memos, journals, and emails and; information on the site including journal articles, dissertations, and information made publicly available on the internet. The results that come from this study have broader implications on how a culture of evaluation can be established when participants who are involved in the process are all engaged (stakeholders, researchers, evaluators) - in the case of this study, a university laboratory school setting.project_summary
The purpose of this study is to see how a culture of evaluation is created within a laboratory school setting. Evaluation theorist Marvin Alkin shared how important context is to evaluation using the following analogy: “in real estate, it’s location, location, location The evaluation counterpart is context, context, context.” Drs. Anne Vo and Christina Christie mentioned that research in evaluation and the practice of evaluation has widely benefited from the knowledge gained from studies that examine when evaluation is used (evaluation use) and when participants involved in the evaluation process are engaged in a specific method driven by theory (stakeholder engagement, prescriptive theory) but evaluation context itself “is a topic that remains under-investigated.”Context, in this study, is defined as the interactions between the administrators, the development director, in partnership with the university’s education department research director and assistant researchers. The results that come from this study have broader implications on how a culture of evaluation can be established when participants who are involved in the process are all engaged (stakeholders, researchers, evaluators) - in the case of this study, a university laboratory school setting. Context is a crucial part of evaluation research and for this reason, Drs. Vo and Christie stress it’s necessary to develop a systematic way to study it. However, an existing systematic design to capture evaluation context has yet to be explored.
This study asks the following research questions: (1) what is the context in which a culture of evaluation is established at a laboratory school? And, (2) how does the school and research team use the results from the alumni survey for the alumni network?
In this proposal, I will conduct a qualitative research study and document the following processes through the following data collection methods:
1) Meetings observations (via Zoom) in which the survey is developed or discussed by Lab School and study researchers. There will be no in-person interactions at this time due to COVID19. These meetings will be audio and video recorded and transcribed via Zoom software using pseudonyms/codes for participants. Participants who will be included in these meeting activities may include the school’s administrators, the development and alumni relations director and team, the research team, and any party involved in the survey development, implementation, and follow-up process.
2) If a participant declines during the individual private oral consent process, the researcher/principal investigator will provide the options to omit information and participation: audio recording via Zoom only, neither video nor audio recording - notes only. The researcher/principal investigator will remove information the participant wishes to leave out.
3) Memos, journals, or emails originating from the principal investigator.
4) I will use observations to conduct a qualitative case study around the process of the development, implementation and follow-up of an alumni survey developed for the Lab School’s Alumni and Development team to document the context of evaluation research and use data collected from documents, artifacts, and others described to support this process. The survey itself targets student alumni who have exited the K-12 system (18+ years old) to learn about their experiences and perceptions of the UCLA Lab School, while developing a larger alumni network for the school. Data results from the survey will not be used in this study – the process around the development, implementation, and follow-up will be the focus of this study.
5) The principal investigator’s personal notes of meetings with the Lab School Alumni and Development team which took place prior to the start of this project from May 2019 to August 2020 will be requested to be used in this study.
6) Collection of publicly available information on the Lab School. This may include journal articles, dissertations, and information made publicly available on the internet such as through available media sources (website, blogs, journals, news articles).
goals
The results that come from this study have broader implications on how a culture of evaluation can be established when participants who are involved in the process are all engaged (stakeholders, researchers, evaluators) - in the case of this study, a university laboratory school setting. An existing systematic design to capture evaluation context has yet to be explored.This study will be presented at conferences (AERA, AEA) and a written document (brief, article, white paper, etc.) will be produced and submitted to a relevant source (journal, school website, university institute website, etc.).
benefits_of_research
Context is a crucial part of evaluation research. There is limited information on an existing systematic design to capture evaluation context. The results from this study will provide useful information about the way in which a culture of evaluation is developed and created in a university laboratory school setting. In addition, this case study will strengthen existing knowledge, such as evaluation literature in research, on ways to provide a systematic approach to capture evaluation context.Expanded knowledge that result from this study will help to inform the Lab School's Alumni Relations and Development Teams as they gain knowledge through the survey development, implementation, and follow-up process of alumni's (ages 18+ years old) impressions on their Lab School experience as well as how to better understand the use of the results from the survey throughout this process.
dissemination/publications
The study seeks to understand the context of evaluation and strengthen the literature on ways to provide a systematic approach to capture evaluation context. The study products that come from this study (papers and presentations) will be submitted to several educational conferences, including AERA (American Educational Research Association), AEA (American Evaluation Association) and CREA (Culturally Relevant Evaluation and Assessment). Findings will be shared with the Lab School’s CONNECT to Research team, school administrators, and development director to inform current practices.numer_of_subjects
6selection_criteria
Participants who will be included in observational activities may include the school’s assistant principal, the development and alumni relations director and team, the research team, and any party involved in the survey development and implementation process. An invitation to participate will take place in two ways, either by email or during a Zoom meeting which will take place in August 2020 and September 2020. See recruitment scripts (direct recruitment and recruitment by message/email).methods
In this proposal, I will be documenting the following processes through the following data collection methods:1) Meetings observations (via Zoom) in which the survey is developed or discussed by Lab School and study researchers. Meetings for observation will take place in a private Zoom room using a unique link. There will be no in-person interactions at this time due to COVID19. These meetings will be audio and video recorded and transcribed via Zoom software using pseudonyms/codes for participants. Participants who will be included in these meeting activities may include the school’s administrators, the development and alumni relations director and team, the research team, and any party involved in the survey development, implementation, and follow-up process.
2) If a participant declines during the individual private oral consent process, the researcher/principal investigator will provide the options to omit information and participation: audio recording via Zoom only, neither video nor audio recording - notes only. The researcher/principal investigator will remove information the participant wishes to leave out.
3) Memos, journals, or emails originating from my own correspondence as the lead researcher on this study with participants.
4) I will use observations to conduct a qualitative case study around the process of the development, implementation and follow-up of an alumni survey developed for the Lab School’s Alumni and Development team to document the context of evaluation research and use data collected from documents, artifacts, and others described to support this process. The survey itself targets student alumni who have exited the K-12 system (18+ years old) to learn about their experiences and perceptions of the UCLA Lab School, while developing a larger alumni network for the school. Data results from the survey will not be used in this study – the process around the development, implementation, and follow-up will be the focus of this study.
5) I will use my personal notes of meetings with the Lab School Alumni and Development team which took place prior to the start of this project from May 2019 to August 2020 and will request from participants to be used in this study.
6) Collection of publicly available information on the Lab School. This may include journal articles, dissertations, and information made publicly available on the internet such as through available media sources (website, blogs, journals, news articles).
For the study’s purposes around the development, implementation, and analysis of the survey, any information collected on the school staff, community members and alumni participants, the principal investigator and the research team will create codes (i.e., pseudonyms). Only myself as the principal investigator and my research team (non-Lab School staff) will have access to the codes. Identifying information will be removed and destroyed at the close of the study.
Interviews will be proposed in the future of this study with individuals involved in the development, implementation, and follow-up of the survey – these interviews will take place when regular Lab School research activities resume during Winter/Spring quarter 2021. I will complete an amendment (with the IRB and the Connect to Research Application) with the interview protocol when given permission and regular research activities resume.
instruments
Arrayinstruments_other
N/Ainstrument_explanations
Observation Protocol: I will be using data an observation protocol to guide the notetaking process. The following protocol above will be used to:• Jot down any notes during onsite observations, meetings, or shadowing of any staff members (online Zoom meetings due to COVID19)
• Observe how assistant principal, coordinators, and research team prepare, implement, and deliver feedback in the Lab School context.
• Capture any observations on context and interaction processes during the length of survey preparation and implementation.
• No personal nor identifying information will be captured in the process of jotting. Researcher will use pseudonyms in the process of notetaking to protect the confidentiality of any subjects.
Survey: Though the results from the survey will not be used in this study, I include it here for reference and context. The survey targets student alumni who have exited the K-12 system (18+ years old) to learn about their experiences and perceptions of the UCLA Lab School, while developing a larger alumni network for the school. The survey is an integral tool for the Alumni and Development team to understand previous alumni’s thoughts. It will be the primary instrument in how conversations are shaped for the researcher to understand the context in which the survey is used by its stakeholders. Results generated from the survey and its analysis will not be directly used in this study – rather the conversations broadly about and around its development, implementation, and follow-up.
justification_of_methods
Though the results from the survey will not be used in this study, it will be the central tool used within the process to understand the interactions around the context of evaluation during its development, implementation, and follow-up. I would like to document the conversations around the stages of the survey’s development, implementation, and follow-up. Gathering notes around this process will help to inform how decisions are made regarding the creation of the survey and the use of survey responses. For this reason, I would like to record meetings. This will primarily take place around the Alumni Relations and Development team, myself as the researcher and other CONNECT Research / UCLA GESIS faculty/staff we will collaborate alongside. Moreover, document collection and analysis will further provide contextual information on the school’s history and context for the background of this study.separate_informed_consent
See attached oral consent form which will be delivered individually to each participant online over Zoom.risk_minimization
Participants will also be asked for their oral consent, privately and individually in a Zoom room over the internet prior to participating in the study. Meetings will take place in a private Zoom room using a unique link due to COVID19. Participating members include those involved in the development, implementation and follow-up of the survey. There will be no in-person interactions at this time. If that changes, interactions will take place in a private room provided by the UCLA Lab School. Upon transcribing any notes, the researcher will use pseudonyms to protect the privacy of the research team and use non-identifying information.deception_debriefing
N/Aconfidentiality_data_storage
The contact information provided, if any, (phone number and/or email) collected during recruitment will be deleted from study records at the conclusion of data collection.For the study’s purposes around the development, implementation, and analysis of the survey, any information collected on the school staff, community members, and alumni participants, the principal investigator and the research team will create codes (i.e., pseudonyms). Only the principal investigator and research team (non-Lab School staff) will have access to the codes. Identifying information will be removed and destroyed at the close of the study.
A secure network server will be used to store data. Any hardcopy data will be locked in a secure place (locked file cabinet). Digital files will be stored on UCLA's BOX cloud storage software, a university-approved, secure method to store and share documents online. All data files will be stripped of personal identifiers and/or the key to the code destroyed.
The contact information provided, if any, (phone number and/or email) collected during recruitment will be deleted from study records at the conclusion of data collection.
For the study’s purposes around the development, implementation, and follow-up of the survey, any information collected on participants, the principal investigator and the research team will create codes (i.e., pseudonyms). Only the principal investigator and research team will have access to the codes. Identifying information will be removed and destroyed at the close of the study.