A significant news story these last many hours has been Google’s refusal to hand over search-related data to the Department of Justice. We can debate the methodological issues associated with the DOJ project for all of the five minutes it takes to realize it’s fundamentally flawed, but I want to focus on the news coverage.

In every press account I initially saw, the headline focused on Google’s refusal, not on the other search engine’s (e.g., MSN and Yahoo) compliance. Is that bias a result of the popularity of Google? Is it reflective of what I perceive to be tendency to support enforcement over privacy issues?

Before asking those types of questions, however, it is important to make sure the facts are straight. Is Google receiving a disproportionate share of the attention of this story?

In support of this is the fact that the media did not pick up on the story until Google made it’s latest refusal, rather than when the other search engines complied. This, however, can be explained away by the fact that the press relies on the government to report the news before they report on it.

A better method would be a large-n study of news headlines. Thanks to the Internet, we can do that with news aggregators. I began with News.Google, the news aggregator I use most often. At first, I searched for each of the major search engine companies in the titles of news stories. Doing so yields these results (at 1210a, January 21):

There are two immediate problems with this procedure. First, it does not only identify news stories about the request for search engine data. While this applies equally to all three companies, today marked the single biggest decline in Google’s stock since its IPO–a development that is more the result of tech’s recent earnings floundering than the DOJ issue. The second problem is that I was using a Google service and it’s possible that the service biases itself.

Before abandoning News.Google, I counted the number of headlines that contained each of the three major search engines AND were related to the DOJ matter. I sorted by date so that articles would not be grouped together based on relevance and used the first three pages for each of the three search results mentioned above. Here are the results:

While doing so, I found a potential problem. Some of the news stories aggregated by Yahoo! News has “Yahoo! News” added to the end of the headline. So some news stories in the Yahoo search were completely unrelated to Yahoo at all, but were still included. That means Yahoo’s total for Measure 01 is probably too high (results were included that are unrelated to Yahoo) and for Measure 02 it is too low (the same number of headlines were checked for content, but Yahoo’s returns had less Yahoo-related articles to count from).

Next step, try again using a generic search term. I choose: “search engine” + justice. Using the first three pages of results, sorted by date, I tallied the number of headlines that had to do with the DOJ issue and included each search engine name. The results:

But this doesn’t address the possibility that each search engine biases results with its name in it. I am not aware of news aggregators that are not run by or based upon the major search engines, and a quick Google search only turned up RSS aggregators. So I test each of the other two search engines to see if we find a common trend among all.

Yahoo offers a more sophisticated news search tool than Google. Interesting. I can search the headline and search the body (among other things). For each of the three major search engines, I search the headlines for the search engine name (e.g., Google) and search “any part of the article” for “justice”. Here are the results:

Now counting the first three pages of date sorted results for “search engine” + justice:

Looks like Google still comes out on top. Now for MSN News. I could not figure out advanced search techniques (their “Learn more about our service” page was a contact form for problems). So I first did a general news search for each of the three major search engines:

Interesting in that they are all closer to each other, but there are too many problems with this method, some of which are outlined above. I then tried what I think is the best measure outlined so far, which is to count the number of times each search engine appears in a headline about the DOJ issue in the first three pages of a search for “search engine + justice. Note that there is no option to sort by date.

While each of these measures has problems, the results suggest that the coverage was skewed toward focusing on Google’s non-compliance rather than the other companies’ compliance. This may be a result of the fact that the event that was being covered involved Google, but this does not explain a lack of earlier or multi-dimensional coverage. Furthermore, I learned some interesting techniques when using news aggregators.