Magic Judges are Obsolete; It's Time to Replace Them With Prediction Markets

Published on April 1st, 2022.

In the late 2000s, Magic judging underwent a revolution due to the cell phone. Previously, we'd have to memorize any rule that might be relevant, or look it up in a rulebook of 200+ pages. This was time-intensive and lead to more guessing and mistakes. The cell phone allowed us to have access to all the tournament documents at our fingertips. It dramatically improved the speed quality of our rulings. Today, Magic judging is poised for a similar if not even bigger revolution: Prediction markets.

In case you've been living under a rock: What are prediction markets? Prediction markets are a revolutionary way to aggregate information and arrive at an informed consensus. Stone age technology like opinion polls can be somewhat better than asking a random person on the street, but they have the obvious problem of letting just any shmuck voice their opinion. If someone is significantly better-informed than the average, their contribution will be drowned out. Can you really expect accurate results from that? Of course not.

Expert opinion is just as flawed; experts are woefully overconfident and are often no better than chance.

Prediction markets improve on both by making people pay to express their opinion, and rewarding them if they were right. This means that people with more information can have an outsized impact on the overall result, and over time the crowd of participants will self-filter for accuracy. More and more organizations are beginning to use them for their decision-making, from Google to Ford to the US Department of Defense. Bringing them to Magic is the obvious next step.

One of our primary jobs as judges is to answer rules questions. This can be hard; Magic is a complicated game, and no one person can remember all the rules, which isn't helped by them changing every few months. We currently attempt to address this problem by forcing judges to go through extensive training programs and testing requirements, which nonetheless result in a pretty dismal accuracy rate. Nearly 50% of surveyed judges at a Grand Prix couldn't correctly explain whether Terrarion lets the player see the drawn card before choosing what color of mana to add.

Prediction markets are a much more elegant solution to this problem. With so much information about Magic's rules publicly available, they should reliably be able to converge on the correct answer. And with the advent of allowing electronic devices at sanctioned tournaments, it's easy to see how this would work. Any time a player in an event has a question, they could just open their Companion app, type in a question, and immediately have the full power of the free market at their fingertips. No more waiting around for a busy judge to find time to answer their call. No more worrying about an incorrect answer losing them the game. No more sore throats from having to yell for a judge, or tired arms from having to hold your hand in the air so that judge can find you. Just a quick look at your phone and then back to the game!

As a proof of concept, here is a prediction market correctly answering a question about protection and the mana value of {X} on the stack. It also arrived at a correct answer to this tricky question about replacement effects application order; I've seen this trip up level 3 judges! Policy questions appear to be no obstacle either; what to do if a player draws an extra card has changed several times over the past few years, but the market arrived at the current answer with no trouble.These preliminary examples did take a few hours before providing an answer, but I'm sure that won't be an issue with increased volume.

Prediction markets would even resolve a long-standing conundrum; what do we do when the official documents don't contain an answer? The best we can do right now is ask the head judge of the tournament to make a decision based on arcane unwritten knowledge, often spawning lengthy arguments over pretty arbitrary decisions. Prediction markets solve all this. Just ask "how would Wizards of the Coast rule this situation?" and go by the market's best guess.A Wizards employee would periodically resolve a few markets at their leisure in order to keep the markets grounded. Similar to human-guided reinforcement learning, they wouldn't need to resolve every market or even most of them. Just a few at random will be enough to incentivize traders to make accurate predictions, and the rest of the markets can resolve to majority opinion.

The improvements offered by prediction markets go much further than rules questions though. One of the guiding principles of tournament policy has been that we don't want to rely on judges to make strategical decisions. Simply put, lots of judges have very little strategical knowledge, and may be worse at figuring out the relevant decisions than the players in the game.Not all of them though. This is why infractions like Missed Trigger and Hidden Card Error ask the opponent to make the decision instead of the judge, despite the additional damage to the game state that can entail.Such as the opponent learning what was in the player's hand.

Prediction markets have ample access to skilled players, so this need not be an issue. Tournament policy can be rewritten to account for this newfound possibility, and become much better at handling situations that require such judgement. No longer would we need to extend rounds past 50 minutes and risk mutually-unwanted draws; just take a picture of the game state and receive an accurate forecast of each player's chances of winning.

Yet another area in which prediction markets could improve tournaments is that of cheating investigations. Investigations are really hard, and even experienced judges often find themselves unsure of whether the given evidence is enough to disqualify a player. This can be a significant source of stress, and judges may worry about having disqualified an innocent player. Prediction markets would allow us to offload the decision to a more reliable and less biased source. The accused player could even participate in their own market, giving them a strong incentive to "tell on themselves" if they were in fact cheating. There's no downside!

This is not even getting into all the benefits for tournament organizers, such as the ability to find out in advance which format will give them the best turnout. From improved ruling accuracy, to lower staffing costs, to no more accusations of partiality or incompetence, prediction markets are clearly the way forward. RulesGuru has taken the first step towards this bright future, and I'm sure the rest of the judge world is soon to follow.