Most Credible VAM Argument Is 0%

At the Shanker Institute, a debate about value-added methods (VAM) for teacher evaluation was detailed on Twitter (@shankerinst), including comments from Tom Kane (Harvard) who makes two comments worth noting:

Kane: Need to move from talking about WHETHER to use value-added and more about HOW to do so. #VAMuses

Shanker Institute ‏@shankerinst

Kane: Nobody arguing for using value-added as 100%, but arguments for 0% do not hold up scrutiny. #VAMuses

Kane’s first point is exactly wrong. In fact, we have left behind the whether long ago, without giving that key question the focus it deserved because, to his second point, the only percentage of VAM we should consider is, in fact, 0%. Why?

  • A fundamental premise in statistics is do not use a metric for some purpose other than the one for which it was designed. Student tests are not designed to measure teacher quality.
  • An ethical problem with VAM is holding one person (teacher) accountable for another person’s (student’s) performance.
  • Campbell’s Law.
  • What is tested is what is taught—the high-stakes accountability era has taught us that the more we focus on testing, the less we ask from teachers and students.
  • The only fair way to implement VAM is to pre- and post-test every student in every class throughout the U.S. This is not justifiable in its cost in either time or money for the outcomes.
  • Using VAM in any way incentivizes each teacher to use her/his students against the outcomes of other teachers’ students, possibly the most ethically damning aspect of VAM.

VAM is a tremendous waste of time and energy on one aspect of schooling (teacher quality) that is, at best, a minor measurable component of student learning. Committing to how to implement VAM also further detracts time and money better spent on the 85-90% of correlated factors in student outcomes—specifically the 60+% associated with conditions beyond the control of the school or the teacher.

For statisticians, VAM is a playground. But for the real-world of education, it is fool’s gold, an inexcusable waste of time and funding.

—–

Common misunderstanding: Teachers do not CAUSE learning. Teachers create opportunities to learn #VAM0%

Comments

  1. Paul Thomas is right.
    VAM is unreliable. Different tests of the same subject often produce different ratings, and the same teacher’s ratings can vary from year to year, sometimes quite a bit. This has been shown in several studies. In other words, it doesn’t work, period. Using VAM for 40% or 50% of a teacher’s rating means that 40 or 50 percent of the rating is useless.

    Sources:
    Different tests produce different ratings: Papay, J. 2010. Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal 47,2.
Vary from year to year: Sass, T. 2008. The stability of value-added measures of teacher quality and implications for teacher compensation policy. Washington DC: CALDER. (National Center for Analysis of Longitudinal Data in Educational Research.) Kane, T. and Staiger, D. 2009. Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER Working Paper No. 14607 http://www.nber.org/papers/w14607.

  2. It has to start somewhere. It just as well be you . When you decide to just say no, say it loudly and others will follow.

  3. SpecialEducator says:

    I am pre-testing third grade special education students this week with a so-called standards-based test cobbled together by harried teachers over the summer. This “data” from the test–most of them failed to get anything correct on this absurdly difficult test in the first two days–will then be compared to their random guesses on the state CCSS exam in April (which, if I’m not mistaken, is field-tested and norm-referenced, an entirely different animal from a teacher-created, criterion-based test). This is done so I can have a number for my Student Learning Outcome (SLO). This number will contribute a large part toward my APPR score and will indicate if I am an effective teacher or not.

    In a bitter twist of irony, I believe I’ve been an ineffective teacher this week despite following marching orders and giving the exams (both this test, developed by our regional BOCES, and our district’s AIMSweb tests). You see, I saw students really trying hard, believing that they had a chance at succeeding on this test. Obviously, someone along the way filled them with a sense of purpose, resiliency, maybe even grit, that propelled them forward through the pages of incomprehensible multiple choice questions.

    Instead of tasting success, however, I watched them put their heads down in frustration. Groan in despair. Literally squeeze their hair until their knuckles turned white. I saw three of my nine students take over an hour on this untimed test (as opposed to the state test in spring), striving to read each question and answer it to the best of their severely limited abilities. I was dying to tell them to stop, to just guess at the answers so we could move on to something meaningful. But besides the fact that this would violate testing protocol, what implicit message does that send about school and their place in it? IN THE FIRST WEEK OF THIRD GRADE, THEY HAVE PROVEN TO THEMSELVES THEY ARE NOT CUT OUT TO SUCCEED. These kids are going to equate these tests with me. Will they think that all of my quizzes and tests, the logical ones I’ll give to assess their learning, are going to be just as random and purposeless? Probably.

    Was I developing character in these students by placing an impossible challenge before them? Grit? No. I was like a big league manager asking a junior varsity pitcher to come in at the top of the ninth and close a game against the Yankees. Failure was the only expected outcome. When the game is rigged, like APPR, and when kids are uselessly and senselessly manipulated now everyday in NYS schools (which they are) for shady political purposes, what choice do educators have? Where do conscientious educators go from here? Or parents for that matter? How do we unite effectively and halt this senseless corporate takeover of schools?

    I wanted a positive start for these students of mine, most of whom are emotionally fragile. I wanted to build a warm, positive atmosphere in my classroom in which I could earn their trust. Instead, on the second, third and fourth days in my classroom, I’m slapping a test down in front of them. And I haven’t actually started teaching any curriculum yet!! I stomped all over their trust. I hope I can earn it back.

    In a different context, in a different decade, the above description would match the definition of educational malpractice.

Trackbacks

  1. […] away from commitments to charter schools, Common Core, next generation tests, TFA, merit pay, and VAM and toward addressing more clearly defined—and evidence-based—problems facing public […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 14,563 other followers

%d bloggers like this: