What does Google think about cloaking?

Summary:

Matt's answer:

We’re going to talk a little bit about cloaking today.

A lot of people have questions about cloaking. What exactly is it? How does Google define it? Why is it high risk behavior?

There’s a lot of HTML documentation. We’ve done a lot of blog posts. But I wanted to do the definitive cloaking video, answer some of those questions and give people a few rules of thumb to make sure that you’re not in a high risk area.

What is cloaking?

Cloaking is essentially showing different content to users than to Googlebot. Imagine that you have a web server right here. And a user comes and asks for a page. You give him some sort of the page. Everybody’s happy. And now, let’s have Googlebot come and ask for a page as well. And you give Googlebot a page. Now, in the vast majority of situations, the same content goes to Googlebot and to users. Everybody’s happy. Cloaking is when you show different content to users than to Googlebot. And it’s definitely high risk. That’s a violation of our quality guidelines.

If you do a search for quality guidelines on Google, you’ll find a list of all the stuff – a lot of auxiliary documentation about how to find out whether you’re in a high risk area. But let’s just talk through this a little bit. Why do we consider cloaking bad, or why does Google not like cloaking? Well, the answer is sort of in the ancient days of search engines, when you’d see a lot of people do really deceptive or misleading things with cloaking. For example, when Googlebot came, the web server that was cloaking might return a page all about cartoons – Disney cartoons, whatever. But when a user came and visited the page, the web server might return something like porn. And so if you do a search for Disney cartoons on Google, you’d get a page that looked like it would be about cartoons, you’d click on it, and then you’d get porn. That’s a hugely bad experience. People complain about it. It’s an awful experience for users.

All types of cloaking are against our quality guidelines

There’s no such thing as white hat cloaking. Certainly, when somebody’s doing something especially deceptive or misleading, that’s when we care the most. That’s when the web spam team really gets involved. But any type of cloaking is against our guidelines.

What are some rules of thumb to sort of save you the trouble or help you stay out of a high risk area?

One way to think about cloaking is, almost take the page, like you Wget it or you cURL it. You somehow fetch it, and you take a hash of that page. So take all the different content and boil it down to one number. And then you pretend to be Googlebot, with a Googlebot user agent. We even have a Fetch as Googlebot feature in Google Webmaster Tools. So you fetch a page as Googlebot, and you hash that page as well. And if those numbers are different, then that could be a little bit tricky. That could be something where you might be in a high risk area. Now pages can be dynamic. You might have things like timestamps; the ads might change, so it’s not a hard and fast rule.

Another simple heuristic to keep in mind is if you were to look through the code of your web server, would you find something that deliberately checks for a user agent of Googlebot specifically or Googlebot’s IP address specifically? Because if you’re doing something very different, or special, or unusual for Googlebot, either its user agent or its IP address, that’s the potential to maybe be showing different content to Googlebot than to users. And that’s the stuff that’s high risk. So keep those kinds of things in mind.

One question we get from a lot of people who are white hat, and don’t want to be involved in cloaking in any way, and want to make sure that they steer clear of high risk areas, is: what about geolocation and mobile user agents – phones and that sort of things? And the good news, the executive sort of summary, is that you don’t really need to worry about that. But let’s talk through exactly why geolocation and handling mobile phones is not cloaking. OK. So until now, we’ve had one user. Now let’s go ahead and say this user is coming from France. And let’s have a completely different user, and let’s say maybe they’re coming from the United Kingdom. In an ideal world, if you have your content available on a .fr domain, or .uk domain, or in different languages, because you’ve gone through the work to translate them, it’s really, really helpful if someone coming from a French IP address gets their content in French. They’re going to be much happier about that. So what geolocation does is whenever a request comes in to the web server, you look at the IP address and you say, ah, this is a French IP address. I’m going to send them the French language version or send them to .fr version of my domain. If someone comes in and their browser language is English, or their IP address is something from America or Canada, something like that, then you say, aha, English is probably the best message, unless they’re coming from the French part of Canada, of course. So what that is doing is you’re making the decision based on the IP address. As long as you’re not making some specific country that Googlebot belongs to, Googlandia or something like that, then you’re not doing something special or different for Googlebot. At least currently, when we’re making this video, Googlebot crawls from the United States. And so you would treat Googlebot just like a visitor from the United States. You’d serve up content in English. And we typically recommend that you treat Googlebot just like a regular desktop browser – Internet Explorer or whatever a very common desktop browser is for your particular site. So geolocation, that is, looking at the IP address and reacting to that, is totally fine, as long as you’re not reacting specifically to the IP address of just Googlebot, just that very narrow range. Instead, you’re looking at what’s the best user experience overall depending on the IP address.

In the same way, if someone now comes in, and let’s say that they’re coming in from a mobile phone, so they’re accessing it via an iPhone or an Android phone. And you can figure out: that is a completely different user agent. It’s got completely different capabilities. It’s totally fine to respond to that user agent and give them a more squeezed version of the website or something that fits better on a smaller screen. Again, the difference is if you’re treating Googlebot like a desktop user, so that user agent doesn’t have anything special or different that you’re doing, then you should be in perfectly fine shape. You’re looking at the capabilities of the mobile phone, you’re returning an appropriately customized page, but you’re not trying to do anything deceptive or misleading. You’re not treating Googlebot really differently, based on its user agent. And you should be fine there.

One last thing I want to mention and this is a little bit of a power user kind of thing – some people are like “OK, I won’t make the distinction based on the exact user agent string or the exact IP address range that Googlebot comes from, but maybe I’ll say check for cookies. And if somebody doesn’t respond to cookies or if they don’t treat JavaScript the same way, then I’ll carve out and I’ll treat that differently”. And the litmus test there is: are you basically using that as an excuse to try to find a way to treat Googlebot differently or try to find some way to segment Googlebot and make it do a completely different thing?

Again, the instinct behind cloaking is: are you treating users the same way as you’re treating Googlebot? We want to score and return roughly the same page that the user is going to see. So we want the end user experience when they click on a Google result to be the same as if they’d just come to the page themselves. That’s why you shouldn’t treat Googlebot differently. Cloaking is a bad experience and it violates our quality guidelines. And that’s why we do pay attention to it. There’s no such thing as white hat cloaking. We really do want to make sure that the page the user sees is the same page that Googlebot saw.

I hope that kind of helps. I hope that explains a little bit about cloaking, some simple rules of thumb. Again, if you get nothing else from this video, basically ask yourself, do I have special code that looks exactly for the user agent Googlebot or the exact IP address of Googlebot and treat it differently somehow? If you treat it just like everybody else, you send it based on geolocation, you look at the user agent phones, that sort of things is fine. When you’re looking for Googlebot specifically, and when you’re doing something different, that’s where you start to get into a high risk area. We’ve got more documentation on our website. I hope that explains a little bit about why we feel the way we do about cloaking, why we take it seriously, and how we look at the overall effect in trying to decide whether something is cloaking.

The end user effect is what we’re ultimately looking at. And so, regardless of your code, if something is served up that’s radically different to Googlebot than to users, that’s something that we’re probably going to be concerned about.

by Matt Cutts - Google's Head of Search Quality Team

Original video: