The Confidence Game
I've written before about the importance of social confidence while judging. In order for players to accept our rulings, we need to act authoritative; speaking clearly and calmly, not tripping over our words, etc.
This is a minor form of deception. Social confidence acts as a decent proxy for epistemic confidence (how sure you truly are that you're actually right), and outwardly faking confidence helps "sell the ruling" to players who may otherwise be skeptical that you really know what you're talking about. I'm not a big fan of deception, but I've made my peace with this form since it's only performed via implication from body language and mannerisms, and is the sort of thing that's generally necessary in order to get along in normal human society.
But of course in any adversarial situation like this, it's advantageous to come out on top. Judges are trusted more if we project confidence, but we can also do our jobs more effectively if we're able to see though other peoples' outward demeanor and correctly evaluate their actions and statements.
A common example of this is when a judge with a more timid personality is making a ruling with a player who has a more forceful personality. It's not uncommon to see the judge get intimidated into doubting their ruling just because the player confidently insists that it must be incorrect. This generally leads to wasted time as that judge has to double-check a ruling that was right all along.
But this isn't limited to players! At an SCG event a few years ago, there was a judge whom I'll call Alice taking a call. Alice was a newer judge, but quite competent. She took a judge call and thought she had the right answer, but wanted to double-check, so she found another nearby judge, Bob. Bob told her that no, the answer was something else. This didn't sound right to Alice, so she asked Bob if he was sure. Bob said "yes, I'm 99% sure this is the right answer". Alice thought to herself "well, that's quite a high certainty, so I must be wrong here", and went and issued Bob's ruling. It turned out that Alice's original answer had been correct, and Bob's had been wrong.
I found this interaction particularly entertaining due to Alice's (frankly quite endearing) naiveté. She had come from a quantitative academic background where people try to represent their beliefs accurately, so when she heard Bob express a numerical credence, she just trusted that he was well-calibrated on this matter and his number was accurate. In reality, people tend to be overconfident, and someone who says they're 99% sure of something is probably going to be wrong much more frequently than 1 time in 100.
The explicit numbers make that example stand out, but this sort of thing is actually quite common when it happens via more subtle cues. My partner has a similarly non-confrontational personality, and she's told me many stories about times where she was disagreeing with another judge about some rules question, and the other judge presented an explanation of their position that changed her mind. Then only much later would she be thinking about it and realize that what the other judge said didn't actually make any sense, and she had just gone with it on instinct because they had presented their explanation with such self-assurance.
As judges get more experienced, we tend get better about this. Deferring to other people's judgement is often the correct thing to do; when you're just starting out, there's lots that you don't know, so if some other judge seems confident in their belief that something is correct, it makes sense to defer to them. As we gain more experience, we start to understand the underlying factors and can judge for ourselves whether the other person has presented a reasonable justification or not.
So learning how to analyze the words that people have actually said for correctness and tell logical reasoning apart from confident nonsense is an important skill for us. And just recently I've discovered my new favorite tool for training judges in this skill: The Nissa chatbot.
This is a chatbot based on GPT-4, designed specifically to answer questions about Magic, including rules questions. You can ask it about any interaction in natural language, and it will look up the oracle text of the cards involved and give you back a detailed answer.
It will, of course, do a terrible job of this. Its answers are usually completely wrong, and when it lucks into the right end result it almost always still explains the details incorrectly. But it speaks with complete confidence as it does so, writing paragraph after paragraph of perfectly grammatical and eloquent English, complete with technical terms from the CR and explanations of those terms, references to rules (which often don't have anything to do with the interaction at hand), and a numeric confidence level that's almost always above 90%. It plays the role of "confident yet incompetent judge" better than any human I've ever met.
So whenever I'm working with a judge or judge candidate on their rules knowledge, I now try to incorporate some of its answers into my training. I'll have the candidate talk to the chatbot and see if they can successfully avoid getting bamboozled by its charisma and identify the specific errors in its answers.
It's also a great tool for self-directed learning. Even if you go in with the knowledge that it's unreliable, it can still be a challenge to spot the more subtle errors. Not to mention that it does sometimes give the right answer, and you should be able to notice when it's done that. Where something like RulesGuru or a practice judge test lets you work on your technical rules knowledge directly, Nissa instead lets you practice the communication skill of comprehending exactly what someone has told you and seeing if it logically holds together.
I hope you have as much fun with it as I have.
