Could The New Google Spider Be Causing Issues With Websites?

In this article, we hope to share with you the many aspects that this important subject has to offer you.

Around the time Google announced “Big Daddy,” there was a new Googlebot nomadic the web. while then I’ve heard stories from clients of weblocates and waitrs free down and previously unindexed content receiving indexed.

I ongoing digging into this and you’d be astounded at what I found out.

First, let’s look at the timeline of covenantings:

To understand the next part of this article, you need to have a clear grasp of the material that has already been presented to you.

In delayed September some clever spider watchers over at Webmasterworld blemished rare Googlebot activity. In detail, it was in this thread: http://www.webmasterworld.com/forum3/25897-9-10.htm that the bot was first reported on. It nervous some posters who thought that perhaps this could be uniform users masquerading as the eminent bot.

Early on it also appeared that the new bot wasn’t obeying the Robots.txt organize. This is the protocol which allows or denies crawling to parts of a weblocate.

Speculation grew on what the new crawler was awaiting dull Cutts revealed a new Google trial records infeature http://www.mattcutts.com/blog/good-magazines/#statement-5293. For those that don’t know, dull Cutts is a elder wangle with Google and one of the few Google employees chatting to us “uniform folk.” This reveal happened in November.

There wasn’t greatly reveal of Big Daddy awaiting early January of this year when dull again blogged about it asking for view. http://www.mattcutts.com/blog/bigdaddy/

greatly view was given on the accuracy of the outcome. There were also those that asked if the Mozilla Googlebot (known as “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” in your visitor fuel) and Big Daddy were allied, but no answer was made.

Now I’m free to open some of my own speculation:

I do in detail judge the two are allied. In detail, I think this new crawler will eventually trade the old crawlers just as Big Daddy will trade the recent records infrastructure. http://www.passagelinkbrokers.com/bfuel/statements/310_0_1_0_C/

Why is this important?

Based on my observations, this crawler may be able to do so greatly more than the old crawler.

For one, it emulates a newer browser. The old bot was based on the Lynx passage based browser. While I’m confident Google added skin as time went on, the central Lynx browser is just that central.

Which explains why Google couldn’t covenant with effects like JavaScript, CSS and flare.

However, with the new spider, built on the Mozilla engine, there are so many possibilities.

Just look at what your Mozilla or Firefox browser can do itself render CSS, read and finish JavaScript and other scripting languages, even emulate other browsers.

But that’s not all.

I’ve talked to a few of my clients and their locates are receiving hammered by this new spider. It has gotten so bad that some of their waitrs have deceased down because of the number of passage from this one spider!

On the desirable feature, I have clients who went from a few hundred thousand indexed pages to over 10 million in just a few weeks! exactly while December, 2005 there’s been a 3500% intensify in indexed pages over an 8 week epoch! Just so you know, this is also the client’s locate that went down because of the enormous number of crawling episode.

But that’s still not all.

I have another client which uses IP recognition to wait content based on a role’s geographic spot. If you live in the US you get American content and pricing; if you live in the UK you get UK content and pricing. As you may assume, the UK, US, Canadian and Australian content is all very related. In detail about the only thing noticeably different is the pricing phase.

This is my fear if the duplicate content gets indexed by Google what will they do? There’s a good hazard that the locate would be penalized or even banned for violation of the webmaster feature guidelines set forward by Google here: http://www.google.com/webmasters/guidelines.html#feature

This is why we implemented IP recognition so that Googlebot, which crawls from US IP addresses only sees one report of the locate.

However, a check of the waitr fuel shows that this new Googlebot has been visiting not only the US content but also the content of the other sections of the locate. artlessly, I required to verify that the IP recognition was effective. It is. This leads me to speculate then; can this browser spoof its spot and/or use a alternate?

think that the browser is smart enough to do some of its own trialing by viewing the locate from manifold IP addresses. If that’s the reason then those who robe locates are free to have evils.

In any reason, from the imperfect observations I’ve made, this new Google both the records infeature and the spider are free to change the way we do effects.

From beginning to end, this article has helped you to learn more about this topic than you probably thought you would ever know.