Entropy is a measure of disorder, given by the amount of information necessary to precisely specify the state of a system. Entropy is important in information theory and statistical physics. We can give a precise mathematical definition of the entropy in probability theory. We will want a couple of preliminary definitions. Fix a probability space ; that is, is a set, and is a probability measure on . If is a measurable subset of , then the surprisal or self-information of (with respect to )...