I was a member of Toastmasters during most of 2023 and 2024. Most people know that Toastmasters is a place where you go to get more comfortable at public speaking. A lesser known aspect is their approach to evaluation.

If you give a speech at Toastmasters, it will be evaluated. This is something semi-formal, meaning that there is a format and a rubric. That makes sense and is probably what you would think it is. What was unexpected to me was that an evaluation is treated like another speech and is evaluated as well (but we stop there—it’s not an infinite game). The evaluation of the evaluation is less formal. It’s usually a few lines during the general evaluation, which needs to cover the entire meeting. When I had to do it, I would try to pick out a line from the evaluation that was worth emulating, to underscore it.

I thought of this while I’ve been going down a rabbit hole trying to learn about LLM evaluations, which also has the concept of evaluating the evaluators. I don’t have much more to say, but just want to leave a link to Hamel Husain’s excellent post: Creating a LLM-as-a-Judge That Drives Business Results, which was the best thing I found on how to improve LLM based features in a product.