Impactranks

I recently heard of Josh Clark's new project impactranks.com. It is basically a coaches' poll. I respect the intention, but I strongly disagree with the method. Does anyone think the B.C.S. does a good job? It is largely based on a coaches' poll. The problem is that coaches do not see enough other teams play (or debate) and end up reflecting the perceived reputation of programs. There are sounder methods to rank teams that reflect actual results: wins, losses, and points. I would very much recommend reading, Who's #1? The Science of Rating and Ranking, by Langville and Meyer, two math professors, for many of the methods. My own method for debate is the weighted win method, explained here and here, in which whom a team beats matters even more than the raw win percentage.

One objection that people might point out is that debaters have so few opponents during the year that there is not enough data to think a mathematical method is any better than a poll. However, I would point out that the same problem exists in college football and professional football, yet mathematical methods based on opponent strength do work reasonably well.

Another objection is that debate has a lot of upsets, because of bad judges, off rounds, or unusual one-time tricks. True, but so does football, and the mathematical methods still work just fine. (In fact, I was even able to calculate an upset rate for college debate -- about 20%.) The other thought that I have had is to use the data on results to evaluate judges at the same time as debaters. Judges who return consistently unusual results (giving wins to worse teams with a lot of regularity) would lower their rating, so a loss from such a judge would not penalize a team by much.

The only difficulty to using these methods is that high school debate records are not stored in a clear format that shows: the two opponents, the judge(s), and the results. But the college records are. So here is my challenge to any reader. Before the N.D.T., I am going to rank all the teams, based solely on the results from the year, and publish that ranking. We will see how many results my rankings correctly predict (given the upset rate of 20%, anything in that neighborhood or better would be excellent). If any reader wants to come up with their own ranking, we will compare the results. You have until March 27th!