moni33 Posted August 14, 2009 CID Share Posted August 14, 2009 When a crawler visits your web site does it visit the pages that are linked together or all the pages that are on your site lets say .mysite.com/top/city.html is not linked to the others will the crawler see it ? Quote Link to comment Share on other sites More sharing options...
zalternate Posted August 14, 2009 CID Share Posted August 14, 2009 For general information. http://www.robotstxt.org/ Not all Robots with obey robot.txt files. Some are a pain in the azz. http://www.robotstxt.org/faq/blockjustbad.html But I have found hidden pages on websites via search engines. Or found an old page that once had a link to it from the main pages. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.