Robots.txt Validator

“Search engines will look in your root domain for a special file named ‘robots.txt’ (http://www.mydomain.com/robots.txt). The file tells the robot (spider)

“Search engines will look in your root domain for a special file named ‘robots.txt’ (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard. The format for the robots.txt file is special. It consists of records. Each record consists of two fields : a User-agent line and one or more Disallow: lines…”


Related Posts:

Tips for Live Sketching in Virtual Meetings

Investing in Innovation: One Day, Every Month

Use Journals to Introduce Culture and Accelerate Onboarding

Date:
November 24, 2016
Author:
XPLANE