Skippy Records » long tail

Misunderstanding the long tail

Dr. Skippy — Tue, 01 Dec 2009 17:23:28 +0000

The Economist blows it! (I don’t get to say that very often.) On page 79 of the Nov 28 – Dec 4 2009 issue, there is an article (a world of hits) that misrepresents and misunderstands the ideas behind Anderson’s Long tail and proceeds for 2 more pages to build a story on the misunderstanding. This is not just useless reporting, writing and thinking, it is damaging and a sad illustration of mathematical and analytic carelessness.

I am sure Anderson and others will jump on the many errors here. I just posted this rant in the comments section of the online version of the article…feel much better now

The mistake seems to be not understanding that long tail statistics come from underlying dynamics. In particular, books and movies are small hits before they are big hits. The difference between the long tail and the big hits (possibly call it “tall head”) is how fast a book or movie becomes a hit and how big of a hit it becomes. It is difficult to propose a dynamic in which the bottom and middle of the distribution grow where the big hits don’t get bigger as well. Anderson never proposes such a fundamental shift in underlying dynamics.

There seems to be a complete disconnect between the statistics and the market dynamics producing the statistics in this article.

To say that the long tail ideas predict the demise of the hit is sloppy (or ignorant?). In fact, as technologies that enable access to the long tail become more efficient and capable, there may be an argument that the outcome will be that big hits get bigger. The tall head may grow taller since all tall head items start out with no following at all and grow from long tail-dom–the place where broad cheap access is changing our access most.

Hits will always be big (seems like I wouldn’t need to write a sentence that dumb on purpose!) and big hits will always make big money. The point of the long tail is that there are now opportunities to make significant money on less popular items (e.g. books and movies in the long tail.)

Tagged: long tail, mathematics, the economist

Day-to-Day Tall Head of URL Exploration

Dr. Skippy — Thu, 04 Sep 2008 00:53:38 +0000

This is the 4th post on the statistics of URL exploration. In the previous three (The Long Tail of URL Exploration, What does the Nth Explorer of the Web Find? and The Tall Head of URL Exploration) I looked at how adding users grows the long tail and tall head of URLs for a single day. Today, the data covers 20 days with a relatively constant population.

To get some idea how the tall head evolves, compare the tall head on day 0 with 19 successive days. The plot below shows the Top 10, Top 50, Top 100, Top 500 and Top 1000 URLs for day zero and the fraction of the top URLs on day zero appearing in the tall head on the nth day.

Figure 1. Red-Top 10 URLs; Blue Top 50 URLs;
Green-Top 100 URLs; Cyan-Top 500 URLs; Yellow-Top
1000 URLs. (URLs ranked by visits).

While the Top 10 and Top 50 URLs show stability day after day, the Top 500 and Top 1000 roll over at a fairly constant rate after day one. The plot can be used to estimate the size of the persistent tall head of URLs for this population and the rate at which the tall head evolves.

First, look for a change in behavior from maintaining a constant fraction of the day-0 URLs to a steady decline from day to day. By this heuristic, estimate the persistent tall head to be between 50 and 100 URLs.

Secondly, to estimate the turnover of the tall head, choose the approximate desired tall head, e.g., the Top 500 URLs (cyan), and look at the slope of the line for days 1-19. (Alternately, choose a timescale for which the tall head should turn over to a given fraction remaining, say, 75%, giving a timescale of approximately 15 days.)

Figure 2. Red-Top 500 URLs; Blue Top 1000 URLs, shown
for comparison; Green-Fit to Top 500 URLs. (Days 1-20,
URLs ranked by visits).

The plot above shows the Top 500 URLs rollover about 0.5% per day from days 1 to 20.

The Tall Head of URL Exploration

Dr. Skippy — Fri, 29 Aug 2008 16:02:24 +0000

In The Long Tail of URL Exploration, I looked at the distribution of URL visits by 102K people in a day. In What does the Nth Explorer of the Web Find?, I looked at how adding users grows the long tail and the number of unique URLs explored. In this 3rd (of 4) post, I look at the how the tall head changes as we add explorers.

The tall head is the short list of sites that get the most visits. We could call it the Top 10, Top 100 or whatever we think is relevant. Later I propose a couple of heuristics for determining tall head membership in real time.

The tall head is made up of URLs that much of the population visits. These are the "winner take all" URLs of user attention–URLs like cnn.com, google.com, or facebook.com. For this reason, we might expect that while the long tail is growing with more unique URLs and the number of URLs with 2-3 hits is growing rapidly, the tall head is relatively stable.

This is the case.

One way to think about how stable the tall head might be is to ask how well a subset of the population predicts membership in the tall head for the entire sample. The data from the last two posts is well-suited to look at this question.

Below is a plot of the accuracy of various subsets of the population (the same subsets we used previously) in predicting the Top 10, Top 20, Top 50 and Top 100 URLs of the entire population. Just over 40% percent of the population predicts the Top 100 URLs with 90% accuracy. The Top 10 are predicted to 90% accuracy by 10% of the population.

Figure. Red-Top 10 URLs by visit; Blue Top 20

URLs by visit; Green-Top50 URLs by visits; Cyan-Top

100 URLs by visits.

The composition of the tall head depends relatively weakly on the subset of the population doing the predicting.

How can I predict the tall head for the day by 9 am in the morning? This is the real-time problem of long tail distributions. The dynamics of the system are that real time Web exploration data appears as a time-ordered list of URLs from whatever users happen to be surfing. This means that a real time heuristic for determining top URLs for the day has to rely on the properties of the time series including a small surfer sample size and recent counts of visits.

Fortunately, by the results illustrated above, a small sample size is a pretty good bet for determining tall head URLs. What we are still missing is metrics or intuition for how the long tail distribution evolves over time.

We do know that for a URL to end up in the tall head, it must be visited by many Web explorers. This means that we can rule out all URLs that are visited by only one or two users. This assumption also leads to a heuristic based on the time between visits–URLs visited by many people should have the same visit/time distribution as the users/time distribution of the entire sample. More specifically, we might guess that if the time between visits has an average near 1 day/number of visits and relatively low variance, it is likely to be in the tall head. A project for a little later…

Next post: How does the composition of the tall head change from day to day?

What does the Nth explorer of the Web find?

Dr. Skippy — Wed, 27 Aug 2008 20:17:59 +0000

In The Long Tail of URL Exploration, I looked at the distribution of URLs and visits. This was on the way to trying to answer questions like:

How much overlap is there between the URLs 10 people visit and those in the 11th person’s click stream?
How about the 100th or 100,000th person? Does the millionth user explore any unique URLs at all?
Can we build a model to answer How many people are required to crawl 10% of the Web?

The second part of the answer is to look at how the model of URLs and visits evolves as we add users. To get samples with of different sizes using the same click stream data set, randomly select a subset of the users and run the analysis from the previous post. Through everyone back into the pot and randomly select a slightly larger set. Repeat.

I reran the model for 3%, 4%, 5%, 7%, 9%, 11%, 15%, 20%, 30%, 40%, 50%, 60%, 70%, 80% and 90% of the overall users in the 1-day data set. The sample sized ranged from 3,000 to 91,200 users. For the entire data set, the average user made 184 URL visits during the day. In the randomly chosen subsets, users made an average of between 181 and 187 URL visits with most of the variation in the smaller sample size as expected.

Do I expect the number of unique URLs be linearly proportional to the number of users? Or if users are visiting many of the same URLs and URLs tend to have the "winner take all" properties we looked at before, we might expect the number of unique URLs added by the 90,000th user to be fewer than the number of unique URLs added by the 1,000th user.

I first plotted the number of unique URLs against the number of users in each sample. The curve looks straight but may be slightly concave downward. It is very subtly. I needed to look at the data in a way the amplified the change over the various subsamples.

Below is a plot of the number of unique URLs/user vs. the number of users. This line is flat if the number of URLs is growing linearly with the number of users.

The blue curve is the best fit to another power function ( f(x)=ax^k ). The first few thousand users are contribute more original URLs (>90 URLs per user) to the sample than the 100,000th (83 URLs). If you are the first explorer of the a new world, all of your discoveries are original; when you are a late comer, your contributions are around the margins. It may be surprising how much original content being explored by the 100,000th explorer.

Does the long tail get relatively longer or shorter? For simplicity, I use the URLs with only one visit to represent the long tail. Then ratio of 1-visit URLs to unique URLs decreases subtly. For the smallest samples size 70.0% of the unique URLs are hit only once; for the overall data set, the ratio is 69.2%. To amplify this change like above, the plot of 1-visit URLs per user is shown below.

At 100,000 users, the long tail is growing at 57 URLs per additional user. The decrease with each additional user is slowing. The blue curve is the best fit to another power function.

If the power function is the best explanation of the underlying dynamics, the number of unique URLs and the long tail both continue to grow no matter how many people are exploring. Since an increasing number of people need to explore to keep the exploration rate constant, the cost of exploration per URL goes up as explorers are added.

Does anything interesting happen in the tall head where the big winners are? That will have to wait for another post.

Is this long tail distribution a power law?

Dr. Skippy — Thu, 11 Oct 2007 15:45:07 +0000

The discussion of power law vs long tail came up on Chris Anderson’s Long Tail blog a couple of days ago.

"Power law or not?" vs "Long-Tail or not?" are separate questions. If I understand Chris’thesis, Long Tail is the idea that there is a significant population in the "not-hit" part of the distribution, usually of low volume in any rank, but continuing out to very high ranks.

The idea that there is a region where the distribution is essentially "scale free" seems like the key concept. If we start there, interesting questions include: Can we characterize this region with a power law? And (my favorite), what are the dynamics of the system where scale matters? This last question is at the core of the economics of the long tail businesses in general. For example, determining how inexpensive we make "find" and "acquire" activities corresponds with the "knee" in the distribution.

Scale-free is a misleading mathematical idea in that nothing in nature is actually scale free for all domains. For example, an absurdity of assuming scale-free in every domain WRT music or movie hits is that anything created has at least 1 fan (i.e. we don’t have an arbitrarily small hit)–this introduces scale and consequently, the region of arbitrarily small hits with less than 1 fan can’t be modeled by a power law. That’s a toy case, but illustrates how much scale matters.

Instead of including all the points in the power law fit, maybe we can look at the points up to the knee in power law model (it looks like x~3) and then try to understand what interesting dynamics shape the knee for x>3 with the assumption that some scaling has been introduced by cost, potential audience size limits, or whatever…