[SEO] Disallow search engine crawling

 SEO    Search engine crawling     robots.txt  

▌Introduction

---

If we have any web pages on internet that we don’t want to be seen, don’t forget to set the disallow rules on robots.txt.

▌robots.txt

---

Here is a robots.txt example with some disallow rules.

Notice that the url is case-sensitive!

User-agent: * Disallow: /CMS Disallow: /cms Disallow: /CN/News Disallow: /ZH/News Disallow: /EN/News

▋IIS tool for SEO

You can install the Search Engine Optimization management tool by Web Platform Installer.

And then you will see the tool in your website’s [Features view].

Click on [View existing rules]

Now we can add disallow or allow rules, and the SEO tool will create robots.txt into the website.

▋How to test robots.txt

There are many online rebots.txt testing tools, my favorite ones are as following.

▋Google Search Console

Enter Google Search Console, we have to [ADD A PROPERTY](add a webisite).

And follow the steps to validate that you are the owner of the website.

Once the validation is done, you can use the robots.txt Tester in Search Console.

▋Technicalseo.com: SEO tool

If you would like a quicker and easier tool,  Technicalseo.com: robots.txt Testing Tool is simple and friendly.

▌Reference

---