Jump to content
Sign in to follow this  
moni33

Crawler

Recommended Posts

For general information.

http://www.robotstxt.org/

Not all Robots with obey robot.txt files. Some are a pain in the azz.

http://www.robotstxt.org/faq/blockjustbad.html

But I have found hidden pages on websites via search engines. Or found an old page that once had a link to it from the main pages.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...