Robots.txt is often a file that exists on the basis directory of each Internet site and can be used to instruct search engines on which directories/documents of the website they are able to crawl and incorporate within their index. If you are expecting them to be a hundred% on shopper https://news.rafeeg.ae