As I laid out in my introductory post, I am laying out my conference tournament predictions in order to compare them to other predictions out there. The two that I know of are Ken Pomery’s Log5 predictions and TeamRankings predictions.
In that first post, I proposed a sum of squared errors measure to score each system. After talking with multiple people much smarter and more well-versed in this area than me, I settled on using a logarithmic scoring rule. One way to grade each system’s predictions would be to apply the log to the “winning” probability of each game (for instance if the winning team was given a 75% chance to win, the score for that game would be the log(.75); if the other team were to win, the score would instead be log(.25)). However, each set of predictions simply gives the probability of each team advancing to each round, so we don’t have individual game probabilities. As a replacement, I decided to grade each team based on the predicted odds that they would go exactly as far as they did. Say Team A won their 1st round and quarterfinal games but lost in the semifinals. If the prediction said they had a 75% to make it to the semifinals and a 50% chance to win in the semifinals, then the chance that they win the quarters but lose the semis is 75% – 50% = 25%. Thus, the score for Team A is log(.25). This double counts games, but it double counts every game, so there shouldn’t be a bias.
That’s enough boring math, let’s take a look at some preliminary results, through tonight’s (Wednesday) games. The higher the score (i.e. the less negative it is), the better.
Total Logarithmic Score
Ken Pomeroy: -152.53
Predict the Madness (me): -152.65
Team Rankings: -152.98
The numbers appear to be very close, closer than I thought they’d be. We’re about half way through the conference tournament games, so we’ll see if this holds throughout the rest of the games. One caveat: Liberty, who won the Big South tournament as an ultimate longshot, was given a 0% chance by my system. Unfortunately, I wasn’t thinking, and I used the results of my simulation (1,000 sims) instead of fully calculating out the probabilities. I went back and estimated what I would have given them to win the whole thing, and put the number at 0.2%. Overall, I should have used this method for all of my predictions as the sim just injects noise into the predictions, but hey, you live and you learn. Taking out that game completely actually leapfrogs my system into 1st place as I lost nearly a full “error” point to each other system in that game alone, by far the biggest single-game difference.
I’ll check back in after this weekend once all tournaments are complete to see how each system did, with maybe a quick update between now and then. Until then, enjoy the games.