Google Robot.txt file
How to check that my robots.txt file is working as expected?
The robots.txt analysis tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed. This tool provides results only for Google user-agents (such as Googlebot). [...]
Tags: How to check that my robots.txt file is working as expe, robot.txt file help. how to create robot.txt
Posted in: Google Robot.txt file | No Comments »
Know in detail about robots.txt file
for more detail visit: http://www.robotstxt.org/
Tags: how to create Robot.txt file
Posted in: Google Robot.txt file | No Comments »