{"id":2104,"date":"2019-12-13T14:50:47","date_gmt":"2019-12-13T14:50:47","guid":{"rendered":"https:\/\/www.seotesteronline.com\/?p=2104"},"modified":"2021-05-19T13:33:53","modified_gmt":"2021-05-19T13:33:53","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/www.seotesteronline.com\/blog\/technical-seo\/robots-txt\/","title":{"rendered":"Guide to file Robots.txt: what it is and why it is so important"},"content":{"rendered":"

In this article, we will explore the role of robots.txt, a small file that can make the difference between scoring a high ranking and languishing on the lowest deeps of the SERP.<\/span><\/p>\n

What is robots.txt<\/h2>\n

Robots.txt’s role is to tell the crawler which pages it can require from your site. <\/span>Beware, the spider can still see them<\/b>. It just does not scan them. If you want to hide a page, you should rely on noindex instructions, as specified by <\/span>Google’s Search Console Guide<\/span><\/a>.<\/span><\/p>\n

So, why do you need a robots.txt file?<\/b> Because <\/span>you can make crawling faster and smoother<\/b>, saving your server from too many crawler requests.<\/span> You can exclude duplicate or not essential pages from the scanning that can hurt your ranking.<\/b><\/p>\n

Where to put robots.txt<\/h2>\n

You have to put the robots.txt file <\/span>inside your website’s main directory<\/b> so that its URL is <\/span>http:\/\/www.mywebsite.com\/robots.txt.<\/span><\/i><\/p>\n

Do not put it elsewhere, or it won’t work.<\/span><\/p>\n

How to create robots.txt file<\/h2>\n

Create a .txt file in your website’s main directory and call it “robots”<\/b>. Remember that you can have only one robots.txt file per site.<\/span><\/p>\n

Create a group<\/b><\/p>\n

Create your <\/span>first group<\/b>. A robots.txt file can have one or more groups.\u00a0<\/span><\/p>\n

Each group has one or more instructions <\/b>(also called rules). Remember to <\/span>use only one instruction for each line.<\/b><\/p>\n

Robots.txt istructions<\/b><\/p>\n

Instructions can be of three types:<\/span><\/p>\n