Entropy, Differential Entropy, and Renyi Entropies

This is a repost of an article from my tech blog www.aplaceofsand.com. It’s primarily notes on the subtleties of differential entropy, but also contains a review of discrete entropy, various entropy-related information quantities such as mutual information, and a listing of various axiomatic formulations.

Here are the notes as a PDF (too involved for mathjax):

Differential Entropy