How to Create a Robots.txt file for your Website
This post is intended to help you understand how to create a robots.txt file for your website. But before we commence, it is important that we understand what Robots.txt file is.
What is Robots.txt?
Robots.txt is basically a file, as the name suggests, for robots. Meaning web crawlers and robots on the web are always on the lookout for this file, as it tells them what to do with the content on your website. Whether or not you want your content to be crawled in order to index them, that logic resides inside your Robots.txt file.
Also known as Robots Exclusion Standard or Protocol, you can use it to block a particular content that you do not wish indexed.
There are two rules on which it governs:
- Rule no. 1:
User-agent: Googlebot Disallow: /funFolder/
- Rule no. 2:
User-agent: * Allow: /
Rule no. 1 means that the user agent called “Googlebot” should not access the folder named “funFolder” in your website or any of its subdirectories.
Rule no. 2, on the other hand means, that all user agents can access your entire site.
NOTE: Full website access is generally by default.
How to Access Robots.txt
Your robots.txt file is located at the root of your site. So let’s say if your site name is:
Your robots.txt file will can be seen by typing:
http://dumbitdude.com/robots.txt
How easy is that?
Now when you open the above mentioned robots.txt link, you will be able to discern the things that are inside in your robots.txt file.
For example clicking open the above link, gave me this info:
How to Create Robots.txt File and Save it on Your Website
Creating a robots.txt file is a piece of cake. Just follow the below steps and you will be fine.
Step 1: Open notepad by pressing Windows + R i.e. opening Run box:
Step 2: If you want Googlebot to allow both images and urls from your website to index, simply copy paste the following in that notepad file.
User-agent: Googlebot Disallow: User-agent: Googlebot-image Disallow:
This is how it would look like in your notepad file:
Step 3: Click on Save as in its File Menu and give a name to the file “robots.txt”.
Step 4: Choose a location where you wish to save the file and then click on the Save button.
Saving Robots.txt File in the Root Directory of your Website
Now comes the time to save this robots.txt file in your website’s root directory.
Step 5: Open your cPanel and login to it using your credentials.
Step 6: Go to File Manager. It must be located under the Files section.
Step 7: Once your File Manager opens, look for your website’s root directory. Check the diagram below:
Step 8: Click on Upload option. It must be there on the top bar.
When you click on the Upload button, it will take you to the File Upload page. This is how it looks like:
Step 9: Click on Select File to open the open dialog box. Then select your robots.txt file and then click on Open again.
Step 10: Click on Go back to link to see the changes.
You will notice that the robots.txt file has successfully begun to show in your File Manager.
Alternate Way to Create Robots.txt File on your Website (Directly using cPanel)
Step 1: Follow Steps 5 to 7 in the aforementioned section.
Step 2: Click on + File button. It will open a new dialog box.
Step 3: In the new dialog box that opens type “robots.txt” in New File Name. Then click on Create New File button.
Step 4: The file will get created. Right click on the created robots.txt file and click on Edit as shown.
Step 5: Click on Edit button
Step 6: When you click on Edit, it will open an editor. It might look something like this.
Step 7: Simply copy paste the following:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
Step 8: Then click on Save changes button.
Creating Robots.txt Using Yoast SEO Plugin
There is yet another alternative to the aforementioned methods. If you are using Yoast SEO plugin on your website, creating a Robots.txt file would be a piece of cake. Here are the steps to do that:
Step 1: Go to your Dashboard, and look for SEO in the menu. Then click on Tools.
Step 2: Click on File Editor in the next screen:
Step 3: Once you have clicked on File Editor, next screen would be your robots.txt file. If it isn’t already there, you need to click on Create “Robots.txt” file.
Simply paste the following if you want Googlebot to allow both your images and urls to index.
Step 4: Once done, click on Save changes to robots.txt.
That should do it!
Robots.txt Tester Tool
You can make use of Robots.txt Tester file provided by Google Webmasters to test if your robots.txt file blocks Google web crawlers from accessing URLs on your site.
Here’s the link from which you can use it:
Is there anything interesting that you would like to add to our How to Create a Robots.txt file for your Website post. Please comment below.
Thank you for taking the time to read this.
