Yahoo's recent paper that examines the possibility of using web search data to predict consumer behavior sparked interest. It's covered everywhere, and Yahoo Search Blog has posted a followup, "What Can Search Predict?", that shares the research team's thoughts on it:
It’s easy to see why search data has the potential to predict: Consumers looking to buy a new camera may search the web to compare models, movie-goers may search for the opening date of a new film or to find movie theaters showing it, and travelers planning a vacation may search for places of interest, airline tickets, or hotel rooms. By aggregating the volume of search queries related to retail, movies, or travel, we might be able to predict collective behavior around economics, culture, or politics.
Their conclusion:
Ultimately, search can be useful in predicting real-world events, not because it is better than other traditional data, but because it is fast, convenient, and offers insight into a wide range of topics.
This article startled me into checking its publication date. The claim is so clearly obvious for the outlined applications (camera purchases, vacation bookings, movie grosses), it seems unreasonable to think nobody in the search business was doing this long ago.
So I, um, Googled.
I found an essay on "trendspotting", suggesting the concept came into vogue in the late '90s, and I found an article citing analysis of AltaVista query logs in 1998 (and other query studies since). But nothing in those two mentioned using search engine data for consumer trends.
Perhaps because that was so hot at the time, folks I worked with in e-commerce in the mid to late 90's assumed this is what search engines were doing with their data—data-mining it and selling the intelligence. Our reasoning was: "Well, if the grocery stores can predict what to stock on the shelves and what coupons to mail, surely the search engines can predict macro/economic/consumer trends, and that data must be incredibly valuable. That must be where they're getting their real revenue..."
But maybe they weren't. A month ago, YCombinator's Paul Graham, in "What Happened to Yahoo", suggested Yahoo was making too much money from ads to bother improving commerce related results with his "Revenue Loop" technology.
Perhaps that lack of revenue innovation held true of all the search engines? At least the idea was out there before this Yahoo research coverage.
This USA Today article from January 2003 opens with the line "Can online surfing habits accurately predict real-world economic behavior?" It goes on to say economists scoffed, but is covering overall online behavior, not just searches.
In August 2009, Google discussed whether search trends could be predicted, and felt they could be, a year ahead. This seems focused on the queries themselves, rather than the real world impact.
Earlier, in April 2009, Google talked about predicting the present with Google Trends. The idea was forecasting the "current" level of activity, and considers "turning points" in economic time series. Google then asks "a million monkeys" to go to their keyboards to work out if this is useful.
Hmm. Maybe Google is just a ruse to collect data for a hedge fund management AI, and we're the million monkeys in front of a million computers giving them the data and supplying the cash with every click.
It has been said that if you put a million monkeys in front of a million computers, you would eventually produce an accurate economic forecast. Let's see how well that theory works ...
Hope Google isn't too mad at Yahoo for tipping their hand.