z3.ca

Inter-rater disagreement is desirable

A few recent reads, including Why Greatness Cannot Be Planned by Kenneth O. Stanley suggest that inter-rater disagreement is a strong signal for something that is potentially worthwhile.

Juries are often required to come to unanimous consensus, but the Rabbinical tradition (see Notes on The Unanimous Verdict According to the Talmud) has the exact opposite philosophy. If consensus is reached in a Sanhedrin trial, it was assumed that the Rabbis involved didn't try hard enough to give the suspect benefit of the doubt. The accused is declared innocent. This is a systematic appeal to engage in devil's advocate thinking (see The tenth man strategy).

Similarly, I am experimenting with hiring people with mixed interview feedback. Show me a bimodal distribution of strong-hire / strong-no-hire and my interest piques.

Against Rotten Tomatoes makes an argument for selecting movies to watch also based on a bimodal distribution. This will surely skew to some great films.

Related in spirit to Clarke's first law

Last edited on Aug 2022