Download this Safelink Theme. Download now!

what is Robots.txt? | how to create a robots.txt file

Robots.txt File: Friends today in this post you will know what Robots.txt File is and how to create Robots.txt File. After this Robots.txt File will add your blogger blog or website, so let's know.

robots.txt,how to create robots.txt file,what is a robots.txt file,robots.txt file,how to write a robots.txt file,what is robots.txt,what is robots.txt file,what is robots.txt in seo,how to create robots.txt file for google,robots.txt seo,add robots.txt to site,how to implement robots.txt file

Robots.txt File is a small text file that remains in the root folder of your site. It tells the Search Engine Bots that the part of the site has to do Crawl and Index and the part of the site does not have to crawl.

You must have seen many times, or it must be happening to you that you do not want to get Category, Tag or Label of your Blog / Website Index in Google but still become Index in Google Search Engine.

And sometimes it happens that we want to index our Article / Post in Google but it is not Index. If your article or post are not having index in google, then it can happen only because of your Robots.txt.

What is Robots.txt?

What is Robot.txt File? Like I mentioned above Robots.txt File is a small Text File that lives in Root Folder. This file is definitely small but its work is very big. Robots.txt File is like a Robot that works the way we command. Just like when you update a New Post Publish or Post on your blog.

So Robots.txt File tells Search Engine that today you have published or updated the new article. From which Search Engine shows your Post in its Search Result by Index. You have just come to know about Robots.txt Flie what its work is. You still need to know a lot.

Like how Robots.txt File is, how to make Robots.txt Flie and how to add Robots.txt File to Blogger. So let's move forward and know what is its benefits and what will happen if Robots.txt File is not added.

What is Robots.txt File?

Robots.txt File is a Text File. What we write or write in this Text File Google Search Engine does the same.

Example:- If you do not want to index the Tag, Category, Label of your Blog / Website in Search Engine, then you will disallow them in Robots.txt from which Search Engine comes to know what to Index and who does not.

Why is it necessary for Robots.txt Website?

Whenever the Crawler of Search Engine comes to our Blog or Website to Crawl or Index the Post Articles, he first sees Robots.txt. And follow whatever is written on Robots.txt.

If you do not get Robots.txt File on your Blog / Website, then the Crawler of Search Engine will index all the things from your site that you do not want to show, which should not be Index and error comes in Google Search Console so that you can have a lot of problem.

So that your site does not have Rank and your site may also be removed from google search engine. Therefore, it is very important to add these Robots.txt File to Blogger.

Advantages of adding Robots.Txt to Blogger

It is very important to add Robots.txt to Blogger, and it gives you a lot of benefits that you can read below-

  • You can tell the search engine from Robots.txt that they should index your blog and don't which things.
  • You can Noindex the Duplicate File or Page of your Website.
  • You can make Category, Tags, Author Page and Comments of your website Noindex.
  • If you do not want to index some File, PDF, Image etc.
  • You can secure it to your website.
  • You do not want to Index Internal Search Results Pages.
  • You can index any post in Search Engine soon by adding Sitemap to Robots.txt File.
  • You can SEO your website by blocking a Low Quality Page from it.
  • You can Noindex any Secure File, Page from this.

How is Robots.Txt File?

If you have not created a Robots.txt file, then you should make it as soon as possible, because it is very important for your Blog / Website. If you do not put it on your blog, you may have to face a lot of problems.

To create a Rotobs.txt file, you will get a lot of websit on the Internet, on which you have to enter the url of your blog and it makes you Robots.txt. That looks like something-

User-agent: Mediapartners-Google

Disallow:

User-agent:

Disallow: /search

Allow: /

Sitemap: https://www.kamalgameti.com/sitemap.xml

How to create Robots.txt File?

If you t worry if you don't know how to make Robots.Txt. I will tell you that you will create robots.txt file and how to apply to your blog. I will give you this code that you can copy and apply it to your blog.

Robots.Txt File code is given below.

User-agent: Mediapartners-Google

Disallow:

User-agent:

Disallow: /search

Allow: /

Sitemap: https://www.kamalgameti.com/sitemap.xml

Note:- You have to copy this code, and instead of where https://www.kamalgameti.com is written, you have to put Url (link) of the Home Page of your Blog / Website. Your Robots.txt File will be created. Now let us know about this Code Syntax.

What are the meaning of Syntax of Robots.txt?

You should understand about the Syntax of Robots.txt, you will get a lot of work going forward.

User - agent: Mediapartners - Google : You can hereby hide the Google AdSense's Code from the User.

User - agent : This code is for the user visiting your website to see what they can see on your website and what they cannot see, you always let it be Disallow because it is very important for the security of your website.

Disallow: /search : If you do not want to show the Keyword of your post, then you always keep it disallow and if you do not want to show one of your posts in Google Search, then by pasting the URL of that post here, you can disallow it so that Search Engine will not index it and it will not appear in Search Results.

Here you can paste the URL of any Post, Page and disallow it and Google will not be able to index it.

Allow : This can show what you want to see to the User and let it always be allowed because it allows Search Engine to Index.

Sitemap : Sitemap gives information about your blog that new posts to Search Engine so that Search Engine shows it in its Search Results by Index it soon.

Sitemap also explains how many posts have been published on your website yet.

Your Website Has Robots.txt File How To Know?

Lives in Robots.txt File Root Folder. You can find it out the Robots.txt File of your Website in the way mentioned below, Google also finds out the way below-

Step 1 : First you have to open browser

Step 2 : Now you have to search by writing /robots.txt in the Last of your Website URL, like- https://hindi.kamalgameti.com/robots.txt

Step 3 : As soon as you search, the Robots.txt file of your website will come in front of you, you can see in the screenshot below.

robots.txt,how to create robots.txt file,what is a robots.txt file,robots.txt file,how to write a robots.txt file,what is robots.txt,what is robots.txt file,what is robots.txt in seo,how to create robots.txt file for google,robots.txt seo,add robots.txt to site,how to implement robots.txt file

Question: Which page of the website should be blocked by Robots.txt File?

Answer:You should definitely Noindex the below things by Robots.txt File

  • Category
  • Tags
  • Comments
  • Author
  • Links
  • Admin

It is very important to make all these things Noindex you. FOR SEO.

You will not be able to do all this correctly in Blogger, but if you are a WordPress User, you can make all this Noindex by Yoast SEO Plugin.

How to Add Robots.Txt to blogspot.com

To add Robots.txt to Blogger you have to follow the steps mentioned below. So that you will be able to easily install Robots.txt in your Blog / Website.-

Robots.txt File Code

User-agent: Mediapartners-Google

Disallow:

User-agent:

Disallow: /search

Allow: /

Sitemap: https://hindi.kamalgameti.com/sitemap.xml
  1. Step: First of all you have to copy this Robots.txt File, and add Link (Url) of your Blog / Website instead of https://www.kamalgameti.com.

  2. Step: Now you have to come to your Blog's Dashboard.

  3. Step: Now you have to click on Settings.
    After that you have to come to Crawlers and indexing.
    After that you have to ON the button of Enable custom robots.txt.

  4. Step: Then click on Custom robots.txt.
    Now a Box will be Open in front of you.
    Copy code to that box is to be pasted, but in the code, you must put the URL of your blog instead of https://www.kamalgameti.com.
    Now you have to click on Save chenges.
robots.txt,how to create robots.txt file,what is a robots.txt file,robots.txt file,how to write a robots.txt file,what is robots.txt,what is robots.txt file,what is robots.txt in seo,how to create robots.txt file for google,robots.txt seo,add robots.txt to site,how to implement robots.txt file

Blog Dashboard> Settings> Crawlers and indexing> Custom Robots.txt> Past Robots.txt File code> Save

Now the Robots.txt File has been added to your Blog / Website and now the Search Engine Crawler will Index there which Robots.txt File will allow him.

Conclusion

I hope you understand everything from the information given to you from this post such as what Robots.Txt File is, how to create Robots.txt File, what are the benefits of Robots.txt and how to add Robots.txt to Blogger. If you have not understood anything, you can comment below and tell. I will definitely reply to you.

What is this post Robots.txt File to you? And how to create Robots.txt File? If you have liked, you do not forget to share this article on social media.

Thank you

Conclusion

💖 I this post i have made a step by step tutorial on what is Robots.txt? | how to create a robots.txt file. I hope you have liked it and Please do share with your friends and follow our blog for more

if you face any problems in code or have any questions please feel free to ask in comments section or Contact us

Labels : Blogger , SEO ,

Post a Comment

Cookies Consent

This website uses cookies to ensure you get the best experience on our website.