Here are instructions for creating and using robots.txt file for websites, to the search engines index the content management of your website
robots.txt is a text file is structured, when the spider (bot, crawler) of the SE (Search engine) on the website to collect the data will robots.txt file to see the instructions in this file.
robots.txt can define each of the different bot SE can vary in each area of the website or website?
Some of the SE bots: Googlebot (Google), Googlebot-Image (Google), Yandex (Russia SE), Bingbot (Bing) / Yahoo Slurp (Yahoo) …
The common syntax of robots.txt file
User-agent: audience acceptance bot Disallow / Allow: URL you want to block / allow
*: Represent all
Example: User-agent: * (It means to accept all types of bot.)
Locks entire site
Blocking a folder and everything in it
Disallow: / wp-admin /
Disallow: / private_file.html
Removing a picture from Google Images
User-agent: Googlebot-Image Disallow: / images / sexy.jpg
Remove all images from Google Images:
User-agent: Googlebot-Image Disallow: /
Blocking any one image, for example. Gif
User-agent: Googlebot Disallow: / *. gif $
Things to avoid in the robots.txt file
– Distinguish case sensitive.
– Do not write balance, lack of white terms.
– Do not insert any character other than the command syntax.
– Each statement should be written on one line.
How to create and placement robots.txt file
– Use notepad or any other program that created the file, then rename the file robots.txt.
– Put in the root directory of the website. (Http://www.tin24h.us/robots.txt)Read more :
- Get royalty free Trend Micro Maximum Security – Knowledge Sharing Blog
- Report: Up to 30 million Indonesians own cryptocurrencies
- How to improve the speed of Wifi when accessing the international house pages – VietBlogDao.Com
- Install and play the game Counter-Strike 1.6 on Android phones
- Instructions to recover deleted data with Undelete Plus 3.0