I just needed to look up something in User Interface Design for Programmers by Joel Spolsky and I was reminded of a great chapter on usability testing (only in the print version). One of the insights is the difference between “usability” and “learnability”. He revisited it in another post.

For casual users, learnability and simplicity are more important than usability and power. In that sentence, by “learnability,” I mean, the ability for novices to figure out how to get tasks done rapidly. By “usability,” I mean only the ability to do tasks in a convenient and ergonomic way without making mistakes and without needing to do repetitive tasks. A data entry system that minimizes keystrokes by prefilling things and automatically jumping from field to field is more usable for experienced users, but it’s harder to learn because it behaves unexpectedly to a novice.

The book is worth picking up if you are planning a formal usability test, just to see what kind of things to look for, and what to expect to accomplish. Keep in mind, that if your testers are seeing the UI for the first time, you are testing learnability, not usability.