Introduction

Information theory: the mathematics of telegraphy, is central to statistical language modelling. It makes the idea of quantitity of information precise and formal. This forms a basis for meaningful comparisons between different systems. And the same goes for different versions of the same system, so you can tell whether some change has improved matters or not.