Long Tail Statistics (Kilkki's Formulation Hides Latent Demand)
The expression “The Long Tail” first started to buzz with the WIRED article of that name by Chris Anderson a couple of years ago. Later, he expanded the article into a book explaining the ideas more fully and giving examples from book sales, music and movies.
The idea is explained in many places. Start with Wikipedia. I won’t try to reproduce a basic explanation.
However, recently, I read a paper by Kalevi Kilkki titled "A practical model for analyzing long tails". In it, Kilkki explains an alternate Long Tail distribution to the traditional Power Law distribtution.
I don’t see Kilkki’s motivation–or rather, I don’t trust it. He seems to be blindly "refining" the distribution to get more precision. I don’t trust this for two reasons (1) precision isn’t going to help prediction or design (without understanding underlying processes, at the very least) and (2) I think his quest to line up the dots with the lines hides important effects. We will see. Finally, I think Kilkki over emphasizes the differences and benefits of his model. To try out my main critique, I built a (rough) Mathematica notebook to look at Kilkki’s model and the traditional power law.
As usual, I generated more questions than answers, but, unless I really misunderstood Kilkki, I don’t see the benefits he is pointing to. I hope you enjoy a quick look at The Long Tail and send comments if you have any thoughts.