.Google.com's John Mueller answered a concern about why Google marks webpages that are prohibited from crawling through robots.txt as well as why the it's safe to disregard the associated Search Console files about those creeps.Robot Web Traffic To Inquiry Specification URLs.The person asking the question recorded that crawlers were actually generating links to non-existent concern guideline URLs (? q= xyz) to pages with noindex meta tags that are also shut out in robots.txt. What motivated the concern is that Google.com is crawling the web links to those pages, getting obstructed by robots.txt (without watching a noindex robotics meta tag) then receiving turned up in Google.com Browse Console as "Indexed, though shut out by robots.txt.".The individual inquired the complying with concern:." However here's the big inquiry: why will Google.com index pages when they can't even observe the information? What's the advantage because?".Google's John Mueller validated that if they can't crawl the webpage they can't see the noindex meta tag. He likewise helps make a fascinating mention of the website: search driver, suggesting to neglect the results since the "typical" consumers will not see those results.He created:." Yes, you're right: if our company can't creep the page, our team can't see the noindex. That said, if our experts can not crawl the webpages, then there is actually certainly not a whole lot for our company to index. Therefore while you might find some of those pages along with a targeted site:- inquiry, the normal consumer won't find them, so I definitely would not bother it. Noindex is additionally great (without robots.txt disallow), it merely indicates the Links will certainly find yourself being actually crawled (and also find yourself in the Explore Console document for crawled/not indexed-- neither of these standings result in concerns to the remainder of the site). The integral part is actually that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the restrictions in using the Website: search progressed search driver for analysis main reasons. Some of those causes is since it is actually certainly not attached to the normal search index, it is actually a separate point completely.Google's John Mueller talked about the web site search driver in 2021:." The short solution is that a website: concern is certainly not indicated to be total, nor utilized for diagnostics reasons.A web site query is actually a particular type of search that confines the outcomes to a particular web site. It's essentially simply words internet site, a colon, and afterwards the site's domain.This inquiry confines the results to a particular website. It's not indicated to be a complete selection of all the pages coming from that site.".2. Noindex tag without making use of a robots.txt is actually fine for these type of scenarios where a crawler is actually linking to non-existent webpages that are getting found out through Googlebot.3. URLs along with the noindex tag will definitely produce a "crawled/not recorded" item in Explore Console and also those will not have an adverse impact on the remainder of the website.Read the concern as well as respond to on LinkedIn:.Why would certainly Google mark pages when they can't even view the web content?Featured Picture through Shutterstock/Krakenimages. com.