What is Robots.txt and How To Create / Make Robots.txt File For Your WordPress Website

Robots.txt File is one of the Most Important File for Every Blogger or User and Now the Question that Arise in the Mind of Every New Bloggers or Newbies that What is this Robots.txt File . If You know about this that’s good but if you don’t than I tell You today How Much Important This Robots.txt file for Your Website.

Robots.txt file is an Important or also a Basic Part of the SEO (Search Engine Optimisation ) and Every Bloggers Aware this Very Important Term . If you are Running a WordPress Website or Blog than this File is must for You. I know that you all Have Submit Your Site in Google Webmaster Tools and that’s good but Robots.txt file has its own importance .Now First I Tell You the Introduction about robots.txt file.

What is Robots.txt File

Robots.txt file is a text file which you have to Upload into You Control Panel (CPanel) and after that the Crawler Crawls Your Website with the Help of this Robots.Txt File. In this File You can Manage which Page of Your Website You Want To Crawl by Crawler and In this File You Can Allow and Disallow the Pages which You Want to Crawl and Which You Don’t Want to Crawl.

In other Words I can also Say that Robots Standard Name is Robots Exclusion Protocol or in short we call them Robots.txt Protocol file and this will Help You To Get Known by the Crawlers or Spider or Web roots that which Pages of Your Website You Want to Get Indexed or Which You Don’t want . These Crawlers used by the Most Search Engines Like Google, Yahoo and Bing.

In short way I tell You that with this Robots.txt file You all Can Index Your Website Pages, Categories and the Best Thing is that You Can Have a Control on Crawlers or Spiders that which Pages of Your Website You Want to Index in Faster Rate. If You are Running a WordPress Blog or Website than this a Like a Key of Closed Door which Have Lots of Best Things Store for You.

Now After this Basic Introduction to Robots.txt file File Now I Love to Tell You that How Robots.txt file will Lead To You in Better Search Engine Ranking and Also Generate Some Best Organic Traffic for Your WordPress Site or Blog. So Just See Below.

Don’t Forget To See This : Best 10 SEO Plugins for WordPress Users 2015

How to Check / Test Robots.txt File

The Robots.txt file always Upload to the Root of Your Website and it looks different from My Website . If you want to Check that How Your robots.txt file Looks than it is very simple you just have to Enter www.yourwebsitename.com/robots.txt But before You Check Your Robots.txt file on your Website First You Have to Make Your robots.txt file.After that Once Your robots.txt file is working Proplerly than Your WordPress Blog or Website Optimize .

After You Successfully Create and Upload Your robots.txt file in the cPanel of your Website than Check this Out and You Can See a Screen Like this .

allinguide.com robots file
allinguide.com robots file


How to Create Robots.txt File for Your Website or Blog

There are so many Ways that You all Can Create or Make robots.txt file for Your WordPress Website or Blog . You Can Automatically Generate Robots.txt file or the Simple and the Best Way to Copy Below the Given Robots.txt File and Upload to Your Control Panel (CPanel ) .

Sitemap: http://www.yoursitename.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/
User-agent: NinjaBot
Allow: /
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /

Fill Your Website Name in Your Site name and after that Save it Call robots.txt and after that Install or Upload this robots.txt file to Your Website Root. So that’s all and Now Your Website or Blog will Index Faster Rate with this File.


Create Your Website or Blog Robots.txt file Immediately and Upload to the Root of Your Website to Gain Popularity in the Search Engines like Google , Yahoo and Bing . This will Index Your Web Pages in Faster Mode and remove Bad Links Automatically .

If You have any Question or queries Regarding this Topic Than You Can Use Comment Box and ask Me Your Question.

Leave a Reply

Your email address will not be published. Required fields are marked *