I've commented before on the sense of taking news articles out of Google's index (there isn't any)...but since Murdoch is planning (or, according to some commentators, is pretending to plan) to take News Corp's content off Google by some time next year the idea is worth another look.
The
problem with trying to re-silo content at tis stage in the game is threefold.
First, online consumers
do not have a problem finding content or finding specific content.
In fact, that's the opposite of the problem we face. Content is dyfunctionally abundant. There is so much of it that the
problem is filtering through the billions of pages/articles/songs/photos/films/games/posts/tweets for something useful, entertaining or relevant.
Take some of the content away and unless it is actually unique you've made
searchers' lives a little bit easier (thanks!).
Second, and relatedly, if I'm looking for commentary on the latest NYTco quarterlies and some of the potentially useful information doesn't show up in my search because it has been deliberately withheld from the engine, I'll never know. People resopond to incentives. Hiding content from me when I search doesn't incentivise me to try any other particular alternative behaviour (I'm going to realise that the article which would have answered all my questions is on the New York Post website but since Google doesn't know about it I have to surf over there and look for it? Just how am I going to know that?).
Third, content in a silo will attract no new readers. Two years ago I didn't read Marginal Revolution, but I found it and now I do. Ditto Unqualified Reservations. Ditto...well, everything in my feedreader. It's an interesting idea to run a paid content business that no-one can find. I wouldn't fancy it myself. Newspapers have enough demographic problems without shutting themselves off to the main potential source of new readers.
Mike Arrington theorises that the end-game is for content-providers to cut deals with other (non-Google) search engines, leaving Google high and dry but the content catalogued and searchable somewhere. Fair enough. Assuming that different providers cut deals with different engines (otherwise it's not a market and we've just swapped the Googlopoly for the Bingopoly, whoop-de-doo) we'd be looking at a potentially massive inconvenience for users with no upside benefits at all as they jumped from engine to engine searching for different content. Happily such a deliberate fragmentation of search ignores the existence of adequate search aggregation tools that pre-emptively obviate the obstacles. Data that can be usefully brought together online generally is, because it can be, because that's how it's most useful (hence indeed, err, Google). Trying at this stage in the game to fragment it all again is no solution, it's just another cycle around while someone knocks up a better aggregator.
Digital channels fragment media. They fragmented music, so now we find songs by the track on iTunes and Spotify instead of buying albums. But where the music industry continues to enjoy inherent commercial upside from its fragmented content (iTunes cost money, Spotify has unskippable ads or subs to pay, ringtones are a goldmine) text content has no such inherent commercial upside as a distributed fragment. Read an article on a newspaper website and you see an ad. Read the same content on Google or in Bloglines and you don't. The problem to solve here is not to stop people finding the stuff and reading it on Google - it's to make money when they see it there, to embed some inherent commercial upside in that experience. RSS banners, universal IntelliTXT, anything that follows the content around and makes money from it wherever it's seen. Hell, even movies manage it with product placement and one day they'll work out how to charge extra for all the millions of times those placements are seen by BitTorrenters.
But take the content off Google and I no longer know it exists, and I'm not going to use three or four different search engines to check every time I have a query. Hide it and it simply isn't there.