.Google.com's John Mueller responded to a concern concerning why Google.com marks webpages that are actually forbidden coming from creeping through robots.txt and why the it is actually secure to overlook the relevant Explore Console reports regarding those crawls.Bot Web Traffic To Inquiry Criterion URLs.The person inquiring the concern recorded that robots were actually making links to non-existent inquiry criterion Links (? q= xyz) to web pages along with noindex meta tags that are likewise blocked out in robots.txt. What motivated the concern is actually that Google is actually crawling the hyperlinks to those webpages, acquiring blocked out through robots.txt (without seeing a noindex robotics meta tag) at that point getting shown up in Google Explore Console as "Indexed, though blocked through robots.txt.".The individual talked to the following inquiry:." Yet here's the major inquiry: why will Google index pages when they can not also observe the material? What is actually the benefit in that?".Google's John Mueller verified that if they can not crawl the web page they can't view the noindex meta tag. He also makes an appealing mention of the internet site: hunt operator, encouraging to disregard the end results due to the fact that the "typical" individuals won't observe those end results.He composed:." Yes, you're appropriate: if our company can not creep the page, we can't see the noindex. That pointed out, if our team can't crawl the web pages, then there's certainly not a whole lot for our team to mark. Therefore while you may view several of those webpages with a targeted website:- question, the common customer won't view them, so I definitely would not bother it. Noindex is actually also fine (without robots.txt disallow), it just indicates the URLs will definitely end up being actually crept (and also find yourself in the Search Console report for crawled/not catalogued-- neither of these conditions trigger concerns to the remainder of the web site). The fundamental part is actually that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the limitations in using the Site: hunt evolved hunt operator for diagnostic causes. One of those explanations is since it's not linked to the routine search mark, it is actually a separate trait completely.Google.com's John Mueller commented on the site hunt operator in 2021:." The brief solution is that a website: concern is actually not indicated to be comprehensive, neither made use of for diagnostics objectives.A web site inquiry is a particular kind of search that limits the results to a certain website. It's essentially simply words web site, a colon, and afterwards the website's domain.This concern confines the results to a details internet site. It's not implied to become a comprehensive collection of all the pages from that internet site.".2. Noindex tag without making use of a robots.txt is fine for these type of circumstances where a crawler is actually linking to non-existent webpages that are obtaining uncovered through Googlebot.3. URLs along with the noindex tag will generate a "crawled/not listed" item in Search Console and also those will not have an unfavorable effect on the rest of the web site.Review the question and also address on LinkedIn:.Why will Google index pages when they can not even view the material?Featured Image by Shutterstock/Krakenimages. com.