# # robots.txt for http://www.wikipedia.org/ and friends # # Please note: There are a lot of pages on this site, and there are # some misbehaved spiders out there that go _way_ too fast. If you're # irresponsible, your access to the site may be blocked. # # advertising-related bots: User-agent: Mediapartners-Google* Disallow: /Pesis2009/Pesis2009/ # Wikipedia work bots: User-agent: IsraBot Disallow: User-agent: Orthogaffe Disallow: # Crawlers that are kind enough to obey, but which we'd rather not have # unless they're feeding search engines. User-agent: UbiCrawler Disallow: /Pesis2009/ User-agent: DOC Disallow: /Pesis2009/ User-agent: Zao Disallow: /Pesis2009/ # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: /Pesis2009/ User-agent: Zealbot Disallow: /Pesis2009/ User-agent: MSIECrawler Disallow: /Pesis2009/ User-agent: SiteSnagger Disallow: /Pesis2009/ User-agent: WebStripper Disallow: /Pesis2009/ User-agent: WebCopier Disallow: /Pesis2009/ User-agent: Fetch Disallow: /Pesis2009/ User-agent: Offline Explorer Disallow: /Pesis2009/ User-agent: Teleport Disallow: /Pesis2009/ User-agent: TeleportPro Disallow: /Pesis2009/ User-agent: WebZIP Disallow: /Pesis2009/ User-agent: linko Disallow: /Pesis2009/ User-agent: HTTrack Disallow: /Pesis2009/ User-agent: Microsoft.URL.Control Disallow: /Pesis2009/ User-agent: Xenu Disallow: /Pesis2009/ User-agent: larbin Disallow: /Pesis2009/ User-agent: libwww Disallow: /Pesis2009/ User-agent: ZyBORG Disallow: /Pesis2009/ User-agent: Download Ninja Disallow: /Pesis2009/