How to Optimize Your Robots.txt File
The robots.txt file is a file that sits at the root level of your website and asks spiders and bots to behave themselves when they’re on your site. You can take a look at it by pointing your browser to https://www.yoursite.com/robots.txt.