Don't optimize one thing but measure another

[anishathalye]

I've seen this in a fair number of papers recently, where one quantity is optimized and another is measured.

Examples of this mismatch include:

• conducting targeted attacks but measuring model accuracy (rather than success rate)
• conducting untargeted attacks to change top-1 classification but measuring top-5 accuracy

A related issue includes comparing two attacks that are not directly comparable, for example:

• conducting targeted attacks with (target = least likely class) and measuring accuracy, and comparing with prior work's results with accuracy on attacks of the form (target = random class)

It might be nice to include some text on this topic. Thoughts?