Impact Pedagogy

Adam Pirie

Understandings and Reflections

*

Program Evaluation Design

Program to Evaluate: Learning Strategies

Program Focus: The Learning Strategies program provides a platform through which a teacher can act as a ‘counsellor, coach, facilitator and advocate’ to the students. The program consists of 30 minutes every day in which teachers provide tutorial assistance, facilitate discussion of school issues and consider positive solutions to academic or behavioural issues that arise with students. A key aspect of the program is that students are able to acquire ‘passes’ in order to access their specific subject teachers in the school, for the purpose of asking questions and receiving assistance. Importantly, the Learning Strategies program also provides time for teachers and students to establish strong connections, which provides teachers with a holistic view of students and encourages the development of student-specific supports and interventions.

Staff: The teaching faculty consists of approximately 45 teachers and administrators, and this is supplemented by approximately 20 support staff members.

Goals: The intended outcomes of the program are multifaceted and interrelated, in that the goals of the program are connected and support each other. The short-term goals of the program are to improve assignment completion rates and overall quality, while ensuring that students are able to seek out assistance from teachers when they struggle with content or simply need clarification.  Enhanced assignment completion will not only act to support learning, but along with test scores will evidence the impacts that the program has had on student learning. The long-term goal of the program is to establish and develop teacher-student relationships which will encourage academic performance and support students’ resiliency. A key program outcome is to support students in making increasingly positive academic choices through this teacher-student mentorship. Specifically, teachers can act to assist students through the direct instruction of academic study skills in addition to the provision of positive behavioural supports. The school’s administration hopes that through this establishment of strong, positive relationships that they will have encouraged the development of a nurturing school environment that students are pleased to play a role in. The importance fostering teacher-student relationships for improving student learning should not be underestimated, as it is supported in the work of many researchers, including John Hattie who has variously estimated the effect size at 0.52 and 0.72. To say it another way, a key goal of the program is to establish and nurture strong teacher-student relationships which have been proven to have tangible positive impacts on student learning.

Resources: Learning Strategies is scheduled during regular school hours, and thus students have access to any classroom resources that are normally available. For example, there are approximately 200 school laptops available on classroom carts, which students are able to use for research and assignment completion. An important resource available to students is the ability to receive ‘passes’ from teachers that they want to meet with. Students then use their ‘pass’ during Learning Strategies time to go and meet with the teacher with whom they would like to work with. Teachers may also suggest that a student take a ‘pass’ if they deem some sort of academic intervention to be necessary. Students will also have access to the Learning Commons, which is a reinvention of the school library. The Learning Commons represents a flexible work space that is both physical and virtual, in that online resources can also be accessed from the school’s homepage. Students can access the physical space at any time during regular school hours, and this provides a space for individual work or group collaboration under the guidance of a teacher-librarian.

Community Demographics: Enrolment in this community high school consists of approximately 700-750 students in grades 10 through 12. The school is located in an agricultural community of approximately 13 000 people, and it serves families living both within the town and the surrounding area.  The school itself is made up of students, teachers, and administrators who are mutually dependant upon each other to ensure that high quality teaching and learning is fostered. Specifically, in order to ensure the success of the Learning Strategies program, students need to be dedicated to using the program time productively while also seeking assistance from teachers as required. The demand on students by the program will certainly be high, so it is essential that teachers provide students with feedback which facilitates their development as self-regulated learners. Teachers also work closely with students to develop positive solutions to any academic or behavioural challenges that they may encounter. Administrators ensure that teachers have the necessary information and resources to support students, such as attendance data and an assessment program which allows teachers to track students progress across classes as marks are posted. While the school administration will take the lead role in ensuring that the program evaluation is rigorous, it is also critical that teachers are students take play an active role in the process so that together all of the shareholders are working towards a meaningful evaluation.

Evaluation Purpose: Through the use of qualitative and quantitative measures, the purpose of this evaluation is to assess the impact of the Learning Strategies program on student learning. That is to say, the purpose of the evaluation is to assess the extent to which the tutorial assistance, mentorship and individualized supports have led to greater academic performance by students.  Additionally, the extent to which a positive school culture is being fostered by the Learning Strategies program must be evaluated, as fostering teacher-student relationships is a key program outcome. The ability to assess whether the learning environment in the school is a positive one will rely upon self-reported data from both students and teachers, and the school’s administration will have the opportunity to act upon issues that may become apparent during this evaluation.

Evaluation Questions:  

  • How does Learning Strategies impact assignment completion rates and the overall quality of completed assignments and projects?
  • What has been the impact of Learning Strategies on student learning as evidenced by current and historical assessment data?
  • How satisfied are stakeholders (teachers, students, administrators) with the program’s effectiveness in supporting improved student academic performance and nurturing a positive school culture?

Program Theory:

screen-shot-2016-10-16-at-4-13-37-pm

Logic Model:

screen-shot-2016-11-17-at-8-34-21-pm

Evaluation Approach: A Participatory Evaluation best facilitates the assessment of the impact of Learning Strategies on student learning. This type of evaluation includes the stakeholders in the evaluation process, and in this way ensures that the process is completed in a way that encourages meaningful evaluation and recommendations for improvement. Specifically in evaluating this particular program, teachers and students are given the opportunity to participate in the data collection and analysis.

The Participatory approach empowers students and teachers to play an active role in evaluating the program which affects them most directly as the primary stakeholders. In this way, teachers and students take ownership of the program and in doing so are motivated to meaningfully work towards its improvement. This approach also has the function of fostering organization learning, which contributes to the continual improvement not only of the program but also of future evaluation iterations. A possible challenge in using this approach is to ensure a clear sense of purpose for participation. It is critical then, that school administration clearly communicate the evaluation’s purpose to stakeholders, especially to students who may most require this sense of clarity.

Data Collection Methods:

Quantitative- The best way in which to assess the program’s impact on student learning will be to collect and analyze assessment data. Final course grades will be compared to historical data before the implementation of program, thus working towards isolating the effect the program has had on students’ academic achievement in the school.  Using the school assessment program, evaluators will be able to monitor changes in assignment grades over a term as individualized interventions take effect. A critical challenge for using assessment data to evaluate Learning Strategies, will be controlling for other factors and improvements that have been made in the school over time. That is, a measurable improvement in student achievement today over 10 years ago would certainly not be due solely to the Learning Strategies program.  It is for this reason, that it will be important to use quantitative data to complement the assessment data in the pursuit of a defined sense of the program’s true impact.

Qualitative- The teachers and students in the school will have a great sense of how well the Learning Strategies program works for them, as they will be experiencing the program everyday. For this reason, surveying these two stakeholder groups would provide valuable evaluation data.  The surveys need not necessarily be expansive, but include both written responses as well as rating scales which would allow for tracking over time. The entire teaching staff will be polled as the number of teachers is relatively low at 45, and thus follow up will be reasonably straight forward. For students on the other hand, a representative sample will be selected which improves logistical concerns while also ensuring good data. The teachers and a simple random sample of students will be contacted on a half-termly basis, thus four times per year, and asked to complete the relevant survey. The data collected through this method will be analyzed and assessed in order to ascertain the extent to which students and teachers believe that the program enhances academic achievement. Satisfaction levels can thus be tracked over time in an attempt to encourage improving performance over time, and for administration to be able to analyze whether improvements made to the program ultimately have the intended effect.  Additionally, the surveys will be useful in determining which aspects of the program have been especially impactful according to students and teachers.

Enhancing Evaluation Use:  Given that this evaluation is designed to be participatory, evaluation use need not necessarily be defined solely as the use of the eventual findings. The evaluation process itself will encourage reflection by both teachers and students, and it is through this reflection that the program can be improved.  Stakeholders will consider which elements have been successful and thus should be continued, while also considering which elements are perhaps less successful and thus require removal and replacement. It is through this process, that stakeholders will benefit from the evaluation use ultimately leading to incremental improvement in the Learning Strategies program.  That being said, the final recommendations will still be of significant value which requires the evaluators and the school administration to consider strategies for enhancing evaluation use.  One such strategy, it to report findings to teachers and students in the form of best-practice scenarios.  This strategy will be be most relevant to the evaluation questions pertaining to improving assignment completion and student achievement, and provides a way to make the findings accessible to all stakeholders in a meaningful way. Teachers and students should also be able to access the data collected, in order for them to play an active role in forming recommendations for improvement. In doing so, the administration can ensure that all of the program’s stakeholders feel a sense of ownership over the program and thus invest more deeply in its improvement.  This investment is especially critical for the Learning Strategies program, as one of its intended outcomes is to develop a positive, nurturing school culture through the development of teacher-student relationships.

Standards for Program Evaluation: 

Utility- Not only have the program stakeholders been identified and had their needs considered, but they have been integrated into the program evaluation process itself. Students, teachers and administrators will participate actively in the evaluation process, in order to ensure that the data collection process and ultimately the findings themselves act to illicit future improvements in the program.

Feasibility- The evaluation has been planned in such a way as to minimize the disruption to the normal teaching and learning activities in the school. Specifically, existing structures such as the school’s assessment and email systems are used to collect data to be analysed. In addition information will be collected on a half-termly basis, which will allow for a significant amount of data to be collected without the collection itself becoming a major burden on teachers and students.

Propriety- It is imperative that the welfare and rights of all stakeholders be protected at all times during the evaluation process. Through the evaluation of the Learning Strategies program, the school’s confidentiality and freedom of information policies will continue to be adhered to. This standard is especially considering the fact that the majority of the shareholders, the students, will be minors during the evaluation process and thus require a high degree of welfare protection.

Accuracy- As outlined above, the collection methods will ensure that the the information being analyzed is reliable and valid. The quantitative and qualitative data will be collected and analyzed systematically, so as to collect technically sound information which ultimately forms the basis of future recommendations.


Collaborative Inquiry Tools

Check out this montage of CI tools that created for my PME 801 course.

%d bloggers like this: