The Robots Exclusion Protocol (REP), more commonly known as robots.txt, is an internet protocol. It is a collection of web standards that are used to regulate the web robot or Search Engine Robot behaviour. This is also used to regulate search engine indexing.
In simpler terms, the robots.txt is a simple text file usually located in the root directory of your website. This robots.txt file tells the search engine (as well as other robots) which areas they are allowed to visit, investigate, and index, along with which areas are prohibited for them.
Please note that you are only allowed to place one robots.txt file for your website and it should only be in the root directory of your website. This is usually just your homepage.
The robot.txt file instructions on each website are respected and followed by major search engines such as Google, Yahoo, Bing, etc. However, keep in mind that these instructions are meant for the search engines to follow, and not at all security measures to keep spambots (that collect emails for spammers) out of your website. To ensure the security of your site, it is advised that you do not rely on the robots.txt file. For this purpose, you should put your files in a protected directory.
What this tool is:
This tool makes the robots.txt for your website for you, so you don't have to figure out the exact coding yourself. To use it, simply fill out the boxes above. Provide your sitemap if you have one, choose whether to allow or deny robots from the many search robots listed, and list any restricted directories that you don't want any robots to be allowed into.
Then click 'create robots.txt' and wait a few seconds. Everything needed for your very own robots.txt will show up in the last box. You may then copy it and paste it into the proper text file on your website. It is much less complex than it may look at first glance, and you do not need to have any coding knowledge to get this done. The tool does all of the proper coding for you in an instant.
A dialogue may pop up on the tool once you click the button, but you can just click away from it, as it does not contain anything important. It is just a part of the tool's process.
