Google Sends Notifications of HTTP/2 Googlebot Crawling

Google announced that it is sending out notices to let publishers know that Google has begun crawling their websites with the advanced HTTP/2 protocol. The notice is only being sent out to those whose sites have been upgraded.

Why HTTP/2

HTTP/2 (also referred to as h2) is a network protocol that servers, browsers and bots can use to transfer data from a server.

HTTP/2 is more efficient than the older HTTP/1.1 protocol and is able to transfer data at faster rates the older protocol.

The benefit for publishers is that this will result in less server load, which means there is a decreased possibility of an error (like a timeout error) when Google is crawling a site at the same time the server is under a heavy load.

An additional upside is that with less strain the site will be able toremain zippy for users that are accessing the website.

The announcement that Google was sending out notices was made over Twitter by Google’s Gary Illyes.

The tweet said:

“just pressed the button to send a large batch of messages to sites which have been opted in to HTTP/2 crawling. if something is not clear, follow the link in the message”

It was accompanied by a screenshot of an example of a notice, showing what it looks like.

Another tweet indicated that Googlebot’s HTTP/2 crawling is slowly coming online, not all at once.

According to Gary:

“the h2 crawler traffic is gradually ramping up, it’s not like you got the message and suddenly all crawl is h2. it may take a few days”

Will Googlebot Crawl all Eligible Sites?

Google will determine whether a site benefits from the new HTTP/2 crawl. If it sees there is no benefit then Google may decide not to use the new HTTP/2 protocol.

According to Google:

“In our evaluations we found little to no benefit for certain sites (for example, those with very low qps) when crawling over h2. Therefore we have decided to switch crawling to h2 only when there’s clear benefit for the site. We’ll continue to evaluate the performance gains and may change our criteria for switching in the future.”

Crawling with HTTP/2 also depends on whether or not your server is set up to handle it. If you don’t know if your site can handle HTTP/2 crawling give it a check at KeyCDN.