When Does a Message Actually Tell You Something?

You wake up and check your phone. Your friend texts: "It's sunny today." How much have you actually learned? If you live in Seattle in November, this message is genuinely surprising — you've learned quite a bit. If you live in Phoenix in July, you've learned almost nothing. You already knew it would be sunny.

Both messages contain the same words, the same "facts." But they don't give you the same amount of information. The technical definition connects information with surprise — it's about how much uncertainty the message removes.

Information measures how much uncertainty gets removed, not just what facts get communicated.

The Uncertainty You Started With

The amount you learn depends on how uncertain you were to begin with. If you live somewhere with completely unpredictable weather — it rains about half the days and is sunny the other half — then checking the forecast genuinely tells you something.

But if it's sunny 99 days out of 100, checking the forecast barely helps. You already had a pretty good idea what it would say.

Same message — "Let's meet at Java Junction" — but completely different amounts of information. In Scenario 1, you learned nothing. In Scenario 2, you went from complete uncertainty to certainty.

Why Rare Events Feel More Informative

If your normally punctual coworker shows up late, that tells you something — maybe they hit traffic, maybe something's wrong. You pay attention. But if your chronically late coworker shows up late again, you barely notice. The unusual event is surprising because it contradicts your expectations.

Rare events carry more information than common events. When something with a 1% chance actually happens, it's way more surprising — more informative — than when something with a 99% chance happens.

Measuring Surprise with Numbers

Your friend is going to text you a single letter, and you need to figure out what it is by asking yes/no questions. The number of questions you need tells you the information content.

When Options Aren't Equal

Real life isn't usually about equally likely options. When you wake up and see it's sunny (50% likely), you've learned some information. When you see it's snowy (5% likely), you've learned a lot. We want our measure to capture this difference.

The unit bit comes from "binary digit" — it relates back to those yes/no questions. One bit is the information you get from one yes/no answer. The logarithm ensures that if two independent things happen, the total information is just the sum.

The Average Surprise: Entropy

Entropy is the average self-information across all possible outcomes, weighted by how often each outcome occurs. It tells you how unpredictable a system is overall — not just how surprising one event is, but how much uncertainty you face on average.

High entropy = high unpredictability = lots of information when you observe the outcome. Low entropy = high predictability = little information when you observe the outcome.

Real-World Application: Text Messages

When you type a message, your phone tries to predict the next letter. Some predictions are easy: after "QU," the next letter is almost certainly a vowel. That's low entropy. But after "TH," the next letter could be E, A, I, O, R, or several others. That's higher entropy.

Your phone's predictive text works better when entropy is low. It's information theory in action — the phone minimizes the information you need to transmit by exploiting patterns in language.

When Your Model Doesn't Match Reality

What if you have a guess about how something works, but your guess is wrong? Imagine you predict your bus will be on time 70% and late 30% of the time, but reality is actually 50/50. You're going to be surprised more often than you expect.

Cross-entropy measures the average surprise you'll experience if you use your predicted probabilities but reality follows the actual probabilities.

KL divergence (Kullback-Leibler divergence) measures the "extra surprise" you experience from using the wrong model. If your model perfectly matched reality, the KL divergence would be zero.

Comparing Two Models

Suppose you're predicting tomorrow's temperature: Cold, Mild, or Hot. You have two competing models, and reality follows its own distribution. Which model is better?

This is exactly how we evaluate machine learning models. When training a spam filter, a recommendation system, or a weather predictor, we calculate cross-entropy between predictions and reality. Lower cross-entropy means better predictions.

What Two Things Tell Us About Each Other

Mutual information quantifies how much knowing one variable tells you about another. Before you know about a student's study habits, you have some uncertainty about their test score. After learning they studied a lot, you have less uncertainty. That reduction is the mutual information.

If two variables are completely independent — like your test score and the weather in Tokyo — the mutual information is zero. If they're perfectly related — like temperature in Celsius and Fahrenheit — the mutual information is maximal.

Putting It All Together: Spam Detection

Email spam detection shows how all these concepts work together in a real system. Your email program faces a question with uncertainty: is this message spam or not?

Why This Matters

Information theory gives us a precise way to talk about uncertainty and learning. Instead of vague statements like "this seems pretty random," we can measure exactly how random something is and exactly how much we learned.

Information is fundamentally about reducing uncertainty. When we receive a message, learn a fact, or observe an outcome, we're not just accumulating data — we're reducing our uncertainty about the world. The amount we learn depends on how uncertain we were before, how surprising the observation is, and how well our mental model matches reality.