In a Facebook discussion 1 stemming from Andy Hollman’s posting of Jacob Bera’s recent piece for the AASB 2 questions of the value of one of AkDEED’s experiments in teacher assessment were raised, and as it seemed to me that the discussion was turning away in part from a focus on what Jacob had to say, I thought I would start a thread focused on that alone.
AkDEED, face with continuing concerns with regard to evidence-based certification and assessment of teachers, decided to look at whether a consensus could be produced regarding appropriate praxis. AkDEED enlisted the assistance of hundreds of Master Teachers who then watched hours of videos of teachers teaching, rating performances on multiple scales. These volunteers then reviewed and discussed what they saw, why they rated the video the way they did, and how any differences might be resolved.
Now, the use of videos as a tool for teacher evaluation is widely accepted (it is in fact a major element of National Certification3 ). However, as anyone familiar with the topic will note, the teacher is free to submit any video the teacher wishes, and may redo the video over and over, and even school their students in performing for the video. So one can see that using video is at best fraught and dubious (yet it is one fundamental to one of Jacob’s arguments, but that is for another discussion …)
What I wish to address here, and what I tried to point out there, are some of the premises underlying any evaluation. There are two matters that are particularly of concern: 1) the manner in which any evaluation rubric is established, and 2) the manner in which evaluators (NBPTS uses the terms “assessors”) are qualified. To put that into rather simple language, can we agree on what good teaching is, and can we identify individuals who can sniff that out?
AKDEED adopted an Aristotelian, rather than a Platonic model. In other words, AkDEED thought they would crowdsource from those who would likely really understand the subtleties of educational praxis. We can arguably contrast that with NBPTS, which self-selected persons to set up a system to find those like themselves. I won’t argue the pros or cons of either methodology here; my focus here is on whether one can expect to obtain a consensus from experts on what those experts are supposed to be doing.
In a sense, AKDEED developed a practical exercise in what can be described as a modified Delphi method, running various prompts repeatedly through a group of experts to see if consensus about what good teaching is could be obtained. Note that this part of the process has little or nothing to do with how the videos were prepared. The focus is on whether what is being viewed is evidence of good teaching. In fact, such videos are used as part of educational instructional across the country.
The results were problematic. There was little agreement on what good teaching was, and some rather heated arguments about what wasn’t. In as much as the responses to videos were arguably more dependent on a teacher’s philosophy than on the evidence that of course made it just as difficult to grapple with the second horn; it was going to be virtually impossible to get any cadre of evaluators to agree on most anything.
Many teachers will assume that they and their colleagues are all extremely competent, but absent broad opportunities for peer review, those views seem more a collection of prejudices than evidence-based conclusions. I have to consider how teachers are “prepared”, and I see little that argues that first year teachers hired from UAA are by virtue of their degree, competent to teach. Whatever your thoughts about the management of ASD, it would appear that ASD has essentially agreed with me for some time. No, I am not saying that all UAA teachers are useless twits; I am saying that we have little evidence to show that they are competent independent professionals.
I have been inordinately lucky in some respects. I have had many teachers observe my teaching and have been able to observe many teachers across numerous subject areas and levels. Personally I think that is invaluable experience and is why I am a proponent of peer review. But then I see teaching like excellent theater, an approach I think not shared by all that many.
But to return to the focus of this reflection, if teachers can’t develop a consensus, then they are either agreeing that anything goes, or that they will knuckle their forehead to some arbitrary authority (who may be a philosopher king, or a tantrumming tyro.) I am not a fan of Doors 2 and 3.
