If the data don't support the prevailing theory, you double-check the data to make sure you're not misinterpreting it, re-run the experiments to make sure they're well controlled, and then if you get the same results...
... you alter your model.
You have no other option. Science is a method for understanding the world based upon logic and evidence. If you accept that as a protocol for knowing the universe, you are dedicating yourself to following where it leads, even if you don't like the conclusions sometimes.
A pity, isn't it, that we can't introduce this approach into other fields?
Like, for example, education. I'm a hard-core linguistics geek; in fact, my master's degree is in Scandinavian linguistics. (Yes, I know, I teach biology. It's a long story.) So one of the things that galls me about education is the fact that we've known for decades that children learn languages better, more easily, and become more fluent the earlier you start them. Kids put into language immersion classes in preschool learn a second language nearly as easily as the first, without tedious memorization of vocabulary lists and conjugations.
And when do most school districts begin language classes? Middle school. Right around the time kids' brains start getting bad at learning languages.
We don't need no scholarly research, educational leaders seem to be saying. We've done it this way for years, and it's working just fine.
So with that scientific-method approach to analysis in mind, let's look at a piece of research regarding the current fad -- high-stakes standardized tests, now being used to evaluate not only students, but teachers, schools, and entire school districts.
[image courtesy of the Wikimedia Commons]
You read that right. Give Tienken information on demographics, ethnic makeup, socioeconomic status, community size, and so on, he can tell you how the school will do on standardized tests before the students actually take them. Tienken writes:
In all, our regression models begin with about 18-21 different indicators. We clean the models and usually end up with 2-4 indicators that demonstrate the greatest predictive power. Then we enter those indicators into an algorithm that most fourth-graders, with an understanding of order or operations, could construct and calculate. Not complicated stuff.I hope you recognize how devastating this is to the claim that standardized tests tell you anything worth knowing about teacher competence. If census data alone predicts student performance, then how are "underperforming" teachers supposed to improve their scores? Tienken's research implies that poor teachers will suddenly become more competent... if they move to a different district.
Our initial work at the 3rd-8th and 11th grade levels in NJ, and grades 3-8 in CT and Iowa have proven fairly accurate. Our prediction accuracy ranges from 62% to over 80% of districts in a state, depending on the grade level and subject tested.
Tienken doesn't mince words about the implications of his study:
The findings from these and other studies raise some serious questions about using results from state standardized tests to rank schools or compare them to other schools in terms of standardized test performance. Our forthcoming results from a series of school level studies at the middle school level produced similar results and raise questions about the appropriateness of using state test results to rank or evaluate teachers or make any potentially life-impacting decisions about educators or children.Now, Tienken isn't saying that teachers make no difference; we all, I think, can attest to the power of a truly skilled teacher in making a difference to a child's life. I had three teachers who stand out as having turned the course of my life in some way -- my high school biology and creative writing teachers, and my first-year college calculus teacher. Each of them engendered in me a passion for learning and a fascination with the topic, such that I looked forward to each and every class and wanted more when I was done.
But the point is, this sort of thing is not measurable with a standardized test. The real value of truly gifted teachers is their capacity for making learning relevant and engaging, making dry academic subjects come to life. And whatever standardized tests are measuring -- a point no one, even the policy wonks at the state and federal Departments of Education, seems to be entirely clear on -- they certainly don't measure that.
So teacher evaluation, astonishingly enough, is best done by a competent administrator, who knows the teacher, the subject, and the students, not by some paper-and-pencil exam. Who'd'a thought.
And it'd be nice if the people in charge would look at Tienken's research, and do a forehead smack, and say, "Wow! We were wrong! Better reconsider how we're applying standardized test scores!" But given that scientific rules of validity and analysis don't seem to apply to education, I have the feeling that the result of Tienken's study will be: nothing. We will almost certainly keep moving down the same road, letting test scores drive more and more decision-making, up to and including teacher salaries and retention.
Can't let a little thing like facts get in the way of educational reform, after all.