Skeptophilia (skep-to-fil-i-a) (n.) - the love of logical thought, skepticism, and thinking critically. Being an exploration of the applications of skeptical thinking to the world at large, with periodic excursions into linguistics, music, politics, cryptozoology, and why people keep seeing the face of Jesus on grilled cheese sandwiches.

Thursday, May 30, 2013

How not to evaluate educators

In his talk "The Surprising Truth About What Motivates Us" (which you can, and should, watch in its entirety here), Daniel Pink describes how the micromanagement and punishment-and-reward system built into the American workplace model simply don't work.  Pink, who worked as an aide to Labor Secretary Robert Reich and who was listed as one of the fifty most influential management thinkers in the world, describes several studies that have findings that are astonishing -- including one that found that in tasks that require engagement and creativity, reward-and-punishment strategies lead to poorer performance.

But what interests me most comes in the middle of the talk, where he describes an Australian software company, Atlassian, that instituted a policy that runs completely counter to the usual business model.  Once a month, every employee at Atlassian is given an entire day to do whatever they'd like to -- in Pink's words, "you can work on whatever you want, with whoever you want, but you have to show us the result in 24 hours."

"In that one day of pure, undiluted autonomy," Pink states, "this has led to a whole array of fixes for existing software, and a whole array of ideas for new products that otherwise would never have emerged."

Autonomy, then -- up to a point, certainly, but far beyond what most businesses are willing to try -- increases productivity, engagement, motivation, and morale.  We do our best, most creative work without someone breathing down our necks.

Which brings me to the latest from the New York State Department of Education.

Some of you may remember from previous posts that the most recent brainstorm from NYSED is that this year the teachers are getting numerical grades.  Yes, folks, in a month or so, I'm going to be getting a report card!

Only fair, you might say, given that I do the same to my students.  And honestly, I have no problem with accountability; I welcome being observed and evaluated.  There is nothing to be gained by a policy that allows inadequate teachers to continue indefinitely in the same job.

The problem is the way the grade is calculated.  Part of it is based on observations by an administrator; but 40% comes from how well students meet "SLOs" -- "Student Learning Objectives" -- based on their performance on standardized tests.  To determine whether the students met their SLOs, an exit exam is administered, and used to see if the student met or exceeded a "target" score based on scores on a pre-assessment.

Let me give you an example from my own class that will illustrate why this is a statistically spurious method.

I teach, amongst other things, AP Biology.  It's a notoriously tough subject, full of technical terms and difficult concepts.  For this class, the "pre-assessment" used is the average of the students' scores on the Regents (Introductory) Biology and Regents Chemistry exams.  Now, the problem is, these are both dramatically easier classes and exams; most of the students who made it to AP Biology scored in the 90s on the Regents Biology exam and at least in the 80s on the Regents Chemistry exam.  So the students walk in with, most of them, a "pre-assessment" score of around 90.

Then, at the end of the year, they hit my cumulative, college-level final.

I can say that almost without exception, everyone scores lower on that exam than they did on their "pre-assessment."  In fact, this year, 12 out of 23 students who took my final did not meet their "state target scores" -- in other words, in New York State's eyes, the students, and I, have failed.  From the point of view of the people at NYSED, it looks like the year my students spent in my class actively made them stupider.

But it gets better.  Because just yesterday, we received a communiqué from Ken Slentz, the Deputy Commissioner of Education in New York State.  The message stated that students who are not eligible to take the science Regents exams because they did not turn in enough labs during the school year -- New York has a requirement that high-school-level science students do a minimum of thirty hours of labs to get credit -- are counted as zeroes against the teacher's score.

You read that right.  A student who was out for medical reasons, and who is on home tutoring and cannot do labs -- which three of my students were, this year, for months at a time -- not only is penalized by not being allowed to take the final exam, but that zero is counted as a failure of the teacher's.  Students who simply disengage, and decide that they have no particular interest in turning in anything, are also counted against the teacher's evaluation score.  "It is the responsibility of the teacher to ensure that all students meet lab requirements so that they are able to sit for the Regents exam," Slentz wrote in an email that was sent to every high school science teacher in New York.  "The Department recommends that districts... create processes that ensure students have opportunities to make up lab requirements."

As a student put it, when I discussed this policy in my Critical Thinking classes, "Wow.  First, teachers have an incentive to give easy final exams.  Now they have an incentive to lie about whether we turned in our lab reports."

In the case of the teachers in my school, I have to say that virtually all of them have responded with indignation.  "I'll be damned if I'll compromise the integrity of my course for some bullshit rule," one said.  "If the state wants to grade me down because of things that are outside of my control, they can knock themselves out."

But what it has done is to destroy morale.  More and more teachers I know are actively looking for other jobs.  One, a friend of long standing who has young children in the school, is looking into finding a way to take her own children out of the public school she teaches in -- a stinging vote of no confidence in the direction public education is going.

My fear, though, is that this trend of turning everything -- students and teachers alike -- into numbers is only beginning.  Micromanaging b-b stackers like Deputy Commissioner Slentz, who evidently don't have the vaguest idea of the reality of classroom teaching, will accomplish nothing by these new mandates but driving those of us who actually care about educating children into other jobs.

As for me: I'm not too far from retirement, and would be eligible for a buy-out (should one be offered) in three years.  I can stick it out that long.  How much longer I'll be able to keep my morale up is another matter.  But after all, given the mediocre grade I'm likely to get on my Report Card this year, it's probably better that I start thinking about getting a job where my evaluation is actually based on my level of performance.


  1. Are you sure about this? I taught ELA in NYS for a number of years -- this was never a factor. Regents no-shows were never part of the calculation. Ever. Is it different for Science? In Science, students must complete a requisite number of labs during the course (what is it, 31?) before they're even allowed to SIT for the exam. Are you talking about the kids who don't qualify to take the exam, or simply the no-shows?

    Please take the time to read my thoughts on my blog, on a VERY similar topic... in this case, the NYS Algebra Regents, the single greatest scam ever perpetrated on public education... Two posts:

    though, to be fair, the first post kind of cannibalizes the second...)

    Maybe we can do some kind of link exchange??

    1. I'm absolutely certain that this is correct. This doesn't mean, of course, that it couldn't change tomorrow; but it comes directly from Ken Slentz, the Deputy Commissioner. The lab requirement is 30 hours -- and yes, I do enough additional to give a comfortable cushion. But I resent the implication that kids who don't complete the lab requirement, for whatever reason, reflect a failure on the part of the teacher. This is a flawed assumption for a variety of reasons.

      I'd be glad to do a link exchange. I'll read your posts this evening!

  2. Devil's Advocate for a second... The theory makes sense... if the teacher could not get the stduent to complete 30 or 31 labs out of 40 or so over the course of an entire academic year, that does signal some failing on the part of the teacher, PERHAPS.

    What I reject is the categorical assumption that anything that negatively impacts a student (or impacts a student's score, since that's all the State really gives a crap about, not the students themselves) is assumed to be DE FACTO the teacher's fault, and the institutionalization of that assumption is DESTROYING public education.

    1. Exactly. It is that last assumption that is the problem.

  3. Wow, having some kind of evaluation of teachers doesn't seem so bad in principle, but they always seem to screw it up so bad.

  4. So, I totally agree on the larger point of this post, but your two specific examples are really more an issue of how your APPR agreement was negotiated locally, than they are the idiocy of the state (though there is plenty of that to go around).

    1. Your pre-assessment process was negotiated in your district among a committee, and your union signed off on it. Here in my district, my AP Bio students don't even take a pre-test, as it is not my SLO course, and if they did, it would have been a collection of questions similar in content and structure to their final. I'm not sure how your district came to decide that the regents grades should be the pre-assessment, but it was done locally, and you should have had input, assuming your union and admin are functional.

    2. The Slentz-lab stupidity is also contingent on your APPR agreement. So, if you have students who are lab ineligible taking an alternate final, then that score is what will be used for SLO scoring.

    Again, not disagreeing with your larger point, but the two examples you cite can be mitigated at the district level. Sorry that it sounds like your district has not been a good shield for the rain of NYSED stupid