How to Create a Robots.txt file for your Website

This post is intended to help you understand how to create a robots.txt file for your website. But before we commence, it is important that we understand what Robots.txt file is.

meme of lisa for Robots.txt file

What is Robots.txt?

Robots.txt is basically a file, as the name suggests, for robots. Meaning web crawlers and robots on the web are always on the lookout for this file, as it tells them what to do with the content on your website. Whether or not you want your content to be crawled in order to index them, that logic resides inside your Robots.txt file.

Also known as Robots Exclusion Standard or Protocol, you can use it to block a particular content that you do not wish indexed.

There are two rules on which it governs:

  • Rule no. 1:
User-agent: Googlebot
Disallow: /funFolder/
  • Rule no. 2:
User-agent: *
Allow: /

Rule no. 1 means that the user agent called “Googlebot” should not access the folder named “funFolder” in your website or any of its subdirectories.

Rule no. 2, on the other hand means, that all user agents can access your entire site.

NOTE: Full website access is generally by default.

How to Access Robots.txt

Your robots.txt file is located at the root of your site. So let’s say if your site name is:

http://dumbitdude.com

Your robots.txt file will can be seen by typing:

http://dumbitdude.com/robots.txt

How easy is that?

Now when you open the above mentioned robots.txt link, you will be able to discern the things that are inside in your robots.txt file.

For example clicking open the above link, gave me this info:

How to Create Robots.txt File and Save it on Your Website

Creating a robots.txt file is a piece of cake. Just follow the below steps and you will be fine.

Step 1: Open notepad by pressing Windows + R i.e. opening Run box:

Step 2: If you want Googlebot to allow both images and urls from your website to index, simply copy paste the following in that notepad file.

User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow: 

This is how it would look like in your notepad file:

How to Create a Robots.txt file for your Website adding data

Step 3: Click on Save as in its File Menu and give a name to the file “robots.txt”.

saving robots.txt file

Step 4: Choose a location where you wish to save the file and then click on the Save button.

Saving Robots.txt File in the Root Directory of your Website

Now comes the time to save this robots.txt file in your website’s root directory.

Step 5: Open your cPanel and login to it using your credentials.

cpanel look for how to create your own website

Step 6: Go to File Manager. It must be located under the Files section.

file manager in cpanel

Step 7: Once your File Manager opens, look for your website’s root directory. Check the diagram below:

opening cpanel file manager

Step 8: Click on Upload option. It must be there on the top bar.

click on upload

When you click on the Upload button, it will take you to the File Upload page. This is how it looks like:

select file of robots.txt

Step 9: Click on Select File to open the open dialog box. Then select your robots.txt file and then click on Open again.

open selected robots.txt

Step 10: Click on Go back to link to see the changes.

go back to root directory

You will notice that the robots.txt file has successfully begun to show in your File Manager.

robots.txt file located in your file manager

Alternate Way to Create Robots.txt File on your Website (Directly using cPanel)

Step 1: Follow Steps 5 to 7 in the aforementioned section.

Step 2: Click on + File button. It will open a new dialog box.

new file in file manager

Step 3: In the new dialog box that opens type “robots.txt” in New File Name. Then click on Create New File button.

create new Robots.txt file for your Website

Step 4: The file will get created. Right click on the created robots.txt file and click on Edit as shown.

click on Edit by right clicking robots.txt file

Step 5: Click on Edit button

click on edit

Step 6: When you click on Edit, it will open an editor. It might look something like this.

empty editor

Step 7: Simply copy paste the following:

 User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:

Step 8: Then click on Save changes button.

pasting googlebot info

Creating Robots.txt Using Yoast SEO Plugin

There is yet another alternative to the aforementioned methods. If you are using Yoast SEO plugin on your website, creating a Robots.txt file would be a piece of cake. Here are the steps to do that:

Step 1: Go to your Dashboard, and look for SEO in the menu. Then click on Tools.

click on Tools in SEO in Yoast SEO

Step 2: Click on File Editor in the next screen:

Click on File Editor

Step 3: Once you have clicked on File Editor, next screen would be your robots.txt file. If it isn’t already there, you need to click on Create “Robots.txt” file.

Simply paste the following if you want Googlebot to allow both your images and urls to index.

How to Create a Robots.txt file for your Website entering data

Step 4: Once done, click on Save changes to robots.txt.

That should do it!

Robots.txt Tester Tool

You can make use of Robots.txt Tester file provided by Google Webmasters to test if your robots.txt file blocks Google web crawlers from accessing URLs on your site.

Here’s the link from which you can use it:

https://www.google.com/webmasters/tools/robots-testing-tool?utm_source=support.google.com/webmasters/&utm_medium=referral&utm_campaign=%206062598

Is there anything interesting that you would like to add to our How to Create a Robots.txt file for your Website post. Please comment below.

Thank you for taking the time to read this.

Scottshak

Poet. Author. Blogger. Screenwriter. Director. Editor. Software Engineer. Author of "Songs of a Ruin" and proud owner of four websites and two production houses. Also, one of the geekiest Test Automation Engineers based in Ahmedabad.

You may also like...

Leave a Reply