Google is on the roll to make its user experience better with yet another announcement. It has stepped up its game by launching Google Extended which is another Google crawler or Google Bot but significantly better. With Google Extended, Google has now given the control to its users for their content to decide whether they want it to help improve Bard or Google AI. So to not let Bard extract any information from your content, you can now mention that in robot.txt with user agent Google Extended and manage your content accordingly with the “NOINDEX” meta tag. Google Extended will inform Google not to crawl those websites and use the content for Bard or Google AI but it will still crawl its other Bots to index the other websites.

What is a crawler?

To cognize the purpose of Google Extended we need to first understand the use of a crawler.

A web crawler or search engine bot scans, indexes, and ranks all the websites based on their content. These Bots are aimed to learn all about the information that is available on websites so it can be retrieved whenever needed to improve the user experience. The websites and information that appear on the screen based on our search are indexed by these bots by scanning the information and contemplating the relevance of that content to our search. 

These Bots are used by all the search engines including Google to improve their working and also utilise this information to enhance Generative AI APIs. As for Google, it uses these bots to improve Bard or Google AI. 

What is robots.txt and NOINDEX meta tag used for?

Robots.txt is a file that informs the crawlers of a search engine about which URLs it can access from a website. It is not to keep Google out of the web page but to protect the sites from overloading with requests. To keep Google out of a page, one can use the “NOINDEX” meta tag which will refrain Google bots from crawling it. The reason to use NOINDEX is to protect sensitive information appearing on search like shopping carts or to avoid some sites appearing on the search that are not ready yet.

Use of Google Extended.

Google Extended intends to simply let the user decide if they want to help improve Bard or Google AI or not. Earlier the crawlers were allowed to retrieve the information for Generative AI APIs but by Using Google Extended users would have complete control and be the masters of their content.