Following up on my last post and trying to answer the question Siva Vaidhyanathan is asking: Should we care about Google’s First Click Free and the possible centralizing force of Google’s policy in this regard?
If there is evidence that Googlebot plus users are treated ‘better’ than others I would say it is problematic. There does not seem to be such evidence and I can think of no obvious incentives.
More advanced, what Google could (try to) do is to ‘contract’ with publishers for special treatment of some users by selecting them. This would allow Google to extract rent from the knowledge about its users, selecting users that are more likely to be turned into a paying customer for the publisher. However, Google does not do so or be willing to do so. This is more the kind of behavior it might be showing on a site like YouTube.
I would love to see evidence if exclusion of websites for certain users happens on a global scale, for instance Google.com using Europeans finding a payment form for a US online newspaper and US citizens finding a First Click Free version. I can imagine Google might make an exception in these cases and not consider this cloaking, but I have to find the answer still.
In some countries (France being an example I believe) there seem to be special deals between Google and publishers in the context of Google News. I have never seen or heard about the details of these contracts. There is an initiative (ACAP) in Europe that has developed a more detailed robots instruction protocol that tries to solve some of the conflicts between search engines and publishers, in my opinion not always in the interest of users. It is based on the idea that robots.txt relates to copyright licensing.
Finally, opaque personalization and geo-targeting of results has in practice done what Carr seems to point to. There is no baseline and there is not one Web for people using Google. Today for instance I found that Google.com shows different results for [mccain] in the US and in the Netherlands. The french fries / potato corporation mccain shows up quite prominently in the Netherlands but not in the U.S., where I am currently located. I plan to post some screen-shots later.