Information Theory

Nyquist (1917) established theoretical limits on how much information could be passed through a telegraph for a given power, and Nyquist (1928) fixed similar limits on the frequency band needed to transmit a given amount of information. Also in 1928 Hartley hit upon a profoundly influential idea: namely that it is in principle possible for any sequence of symbols to be generated either by a sender acting deliberately or as the chance outcome of a sequence of random events. He defined the information content of a message as the logarithm of the number of messages which might have occurred. We will see later in much more detail how and why logarithms get into this story. The key idea for the moment is that it is useful to think of signals as arising from random activities, and to quantify the likelihood that the observed signal arose from the stated model.