IMPLEMENT: Evaluation
Participant evaluations of a PD experience can provide important feedback for the teaching artist.
DEFINITION: What is Evaluation in an arts integration PD experience?
Evaluations often take the form of short surveys that participants complete orally or in writing at the end of a PD experience. They may include both quantitative and qualitative portions, can be organized around learning targets, elicit a personal response, and make room for both directed and open-ended reactions. Written evaluations are usually completed anonymously to encourage an honest critique of the PD experience.
PURPOSE: Why is Evaluation important to an arts integration PD experience?
Participant evaluations of a PD experience can provide important feedback for the teaching artist. Although the teaching artist may have an intuitive read of how the classroom teachers responded to the PD experience, these impressions are often subjective and lacking in actionable specifics. They may also be misleading, incomplete or evanescent. Evaluations, on the other hand, give the teaching artist a chance to hear from each participant. They give the participating classroom teachers a chance to communicate what worked and did not work, what resonated, challenged or confused; and they provide a means by which participants can influence both the content and approach of future offerings of the PD experience.
CHECKLIST REVIEW
✓ What methods will insure complete and candid feedback?
✓ What kinds of evaluation questions will provide yo with actionable feedback to improve future offerings?
✓ How will you analyze and find significant themes within the feedback you receive?
✓ What short-term evaluation questions might help improve an ongoing PD experience?
✓ In what other ways, or for what other audiences, might this feedback be useful?
Our PD instructor did a good job giving only positive feedback which helped everyone feel more comfortable and willing to share their thoughts. This was a helpful reminder for me as a teacher to foster an environment where students feel welcomed and are excited to share their ideas.
One of the most helpful parts of the course is the brief online videos that show the instructors facilitating the strategies. I felt confident implementing the strategies after viewing the videos.
-Classroom Teacher (s)
IN PRACTICE: How might Evaluation be applied to an arts integration PD experience?
Content and Format: It is useful to organize evaluations around goals of the training. For example, Carrying Culture: Micronesia is a three-day course. It has the following learning targets. Teachers will …
- KNOW culturally responsive and culturally sustaining teaching approaches and techniques.
- BE ABLE TO integrate art into their teaching.
- BE ABLE TO facilitate active -learning strategies.
- APPRECIATE and better understand the perspectives of Micronesian students and families.
- APPRECIATE their own skills as artists.
These targets are shared with participants at the beginning of the workshop. Classroom teachers then revisit these targets in their end-of-workshop evaluations, where they are asked to rate on a Likert scale (a rating scale in which the respondent indicates the degree to which they agree or disagree with a statement) the extent to which the workshop accomplished each goal. Starting with the learning targets helps clarify the purpose of the workshop for participants, and provides a clear basis for the teaching artist in gauging the success of the experience. Using a Likert scale helps the teaching artist quantify the results and make comparisons of the relative strengths of different aspects of the program.
Arts Discipline Examples
Click on an arts discipline to view example.
Dance
At the end of a 3-hour PD workshop, participants walk throughout the space until the facilitator prompts them to stand “back-to-back” with a partner. Together, the pair discusses: “What did you learn or discover during this workshop?” They return to their back-to-back positions and develop a full-body pneumonic gesture to reflect something that arose during their talk. They practice that gesture as they walk throughout the space again.
The facilitator calls again, “back-to-back” and each participant stands with a new partner. This time the facilitator asks, “What is something you found difficult or challenging during this workshop?” Again the partners discuss the prompts, return to their back-to-back positions, and develop a full-body pneumonic gesture to reflect what they discussed. They practice their two gestures as they walk throughout the space again.
For a third time, the facilitator calls, “back-to-back” and each participant stands with a new partner. This time the facilitator asks, “What do you plan to use from this workshop when you return to your classroom?” Again the partners discuss the prompts, return to their back-to-back positions, and develop a full-body pneumonic gesture to reflect what they discussed. They practice all three gestures as they walk throughout the space again.
The full group gathers in a large circle, with music playing. The facilitator invites participants to enter the circle at any time, dancing their sequence of three gestures, moving in and then back out to the perimeter.
Finally, on the written evaluation, the classroom teachers respond in writing to all three prompts:
- What did you learn or discover during this workshop?
- What is something you found difficult or challenging during this workshop?
- What do you plan to use from this workshop when you return to your classroom?
Drama
At the conclusion of a three hour PD experience, a short time is set aside for classroom teachers to offer brief answers to the following questions, to help the teaching artist understand the immediate impact of the experience on the participating teachers. They respond within the whole group setting.
- Which drama strategy will you attempt first? Which strategy might you not use (and why)?
- Which parts of the workshop (e.g. experience, facilitation, assessment discussion) were most helpful in this workshop?
- What is something that you would have liked more of?
- What are you most concerned about as you consider implementing the strategy?
At the end of the workshop, teachers fill out a Post-Workshop Questionnaire to provide measurable ratings on specific aspects of the workshop experience.
Music
At the completion of a 1-hour PD experience intended to prepare classroom teachers for a Music and Math residency, the teaching artist requests feedback using a “plus/delta” chart. A “plus/delta” chart is a simple tool to solicit a wide range of feedback from the classroom teachers. A chart with a line drawn down the middle has two categories: Plus (what worked?) and Delta (what could be improved?). The teaching artist offers some areas that would be useful to address such as:
- Pace of the workshop
- Handout quality
- Use of technology
- Thoroughness of instruction in strategies.
- Sequence of learning activities.
- Relevance to standards
Any other thoughts are also welcome.
Participants in the PD experience take a few minutes to write their feedback onto post-it notes and then place the post-its onto the larger chart. Afterward, the teaching artist organizes and reflects on the feedback and considers how to adjust design and instruction for the next workshop in a process of continual improvement.
Visual Arts
This full-day workshop is intended to help classroom teachers: a) become more familiar and comfortable with the museum; b) develop strategies, aligned with their curriculum, for engaging with art at the museum and in their classrooms; and c) increase their skills in drawing and bolster their confidence as artists.
The end-of-workshop evaluation asks specifically about these goals, but also includes open-ended questions to evoke further explanation. The teaching artist collects the completed forms, summarizes the quantitative data, and continues to experiment with different approaches in order to improve the areas that indicate the highest need.
In addition to written evaluations, an oral debrief during the workshop gives participants an opportunity to elaborate on their impressions and can allow the facilitator to ask follow-up questions. To encourage fuller participation in a whole-group debrief, the facilitator may ask participants first to respond to questions in a pair-share, and then ask if anyone wants to share out with the whole group. This approach provides a more sheltered setting for participants to work through their thoughts initially with a partner, as well as the chance to build on the observations and opinions of others. An oral debrief can be used at the end of a PD experience, or partway through to help the facilitator make adjustments along the way. This combination of written and oral evaluations can give the teaching artist quantifiable, representative and nuanced feedback on the efficacy, clarity and relevance of the professional learning experience.
Analysis: Analysis is critical for extracting meaningful insights from classroom teacher evaluations. For the quantitative section, a first step is calculating the mean score for each question. This will give you a general sense of which parts of your training are hitting the mark and where you might need to make changes for the participants to better achieve the learning goals. In addition to allowing you to compare the relative strengths of a given workshop, calculating the mean for each question also allows you to make comparisons between workshops.
If you have a mean score of 4.4 in one workshop, but a mean of 3.2 in a later workshop, it might be worth considering what variables were responsible for this discrepancy. Are there clues in the written responses about why participants in the later workshop had a more neutral response?
Another helpful measure is to calculate the standard deviation for each question. This will give you a sense of the variation in responses for a given question. If the standard deviation is high, it might be worth exploring why participants responded so differently to the same question.
Although not as straightforward, it can also be helpful to try to quantify information from the written responses. As you read through the responses, certain categories or trends may emerge. Tallying these common responses and summarizing your take-aways based on these trends can give you concrete ways to improve a training. Here is an example of one of these trends from the Carrying Culture: Micronesia workshop:
Three participants commented that the second day of the workshop was too sedentary. This is a problem that I was aware of as I put together the program, but was difficult to avoid given the availability of different presenters. By scheduling presenters further in advance, I should be able to space them out better in the future.
By compiling an overall summary of your workshop or other training, you will be able to make visible, significant information to help you improve your practice. This sort of summary can also be a valuable resource as you seek funding and approval for future efforts. A summary might include the following items:
- A brief synopsis of the workshop activities and learning goals
- Photos (participants in action and samples of participant art)
- Results from Likert scale questions (including the mean and standard deviation)
- Transcription of all written responses organized by question
- Summary of significant trends from combination of written responses and Likert scale questions, with suggestions for next steps based on these trends
- Samples from teaching portfolios when applicable
Although this sort of analysis may be time consuming, it can provide a sound basis for improving practice.