Bad Bots: cosa sono e come bloccarli

In questo articolo vedremo di spiegare cosa sono i Bad Bots, perché possono essere rischiosi per il nostro sito web ed in che modo possiamo escluderne e/o limitarne l’accesso alle nostre pagine.

Premessa: Good Bots e Bad Bots

Assolutamente preziosi per la SEO del nostro sito sono i cosiddetti bot (o spider) cioè quei software che, automaticamente, scansionano il nostro sito per comprenderne il contenuto e consentirne l’indicizzazione sui motori di ricerca.

Non tutti i bot, tuttavia, appartengono ai motori di ricerca e non tutti sono "buoni", anzi la maggior parte dei bot che scandagliano la Rete rientrano nella definizione di bad-bots nel senso che spiegheremo tra poco.

Prima di parlare di Bad Bots, tuttavia, è opportuno fare alcune premesse e proporre al lettore una classificazione che gli consenta di chiarirsi le idee sull’argomento.

Cos’è un bot?

Il termine "bot" è la contrazione di "robot" e sta ad indicare un software che accedere alla Rete ed effettua operazioni simili a quelle compiute dagli utenti umani.

Relativamente al nostro ambito, i bot sono utilizzati per scansionare i siti web al fine prevalente di comprenderne il contenuto e/o di prelevarne informazioni.

Bot e motori di ricerca

Ad oggi il bot più importante di tutti è sicuramente GoogleBot, ma non vanno dimenticati anche BingBot, Yahoo! Slurp e molti altri. Grazie a questi bot le pagine del nostro sito vengono incluse nei risultati dei principali motori di ricerca. Attraverso i bot, infatti, i motori di ricerca riescono a capire il contenuto e la struttura del nostro sito e, di conseguenza, sono in grado di suggerire le nostre pagine agli utenti che fanno ricerche pertinenti ai contenuti da noi trattati.

Questi bot sono buoni e dobbiamo essere felici di ogni loro passaggio sulle nostre pagine.

Vediamo di seguito i principali bot dei motori di ricerca:

• Google
  • Googlebot
  • Googlebot-Image
  • Googlebot-News
  • Googlebot-video
• Bing (Microsoft)
  • Bingbot
  • MSNBot
  • MSNBot-Media
  • BingPreview
• Yahoo!
  • Yahoo! Slurp

Altri Bot "utili"

Tra i bot utili rientrano, certamente, quelli utilizzati da Google e dagli altri fornitori di annunci pubblicitari. Questi bot sono molto importanti per tutti i siti che pubblicano annunci pubblicitari mediante network come Google AdSense.

Lo scopo di questi bot è quello di offrire annunci pubblicitari pertinenti alle pagine web ed anche di verificare la qualità di contenuti e annunci in determinati contesti.

I principali bot di questa categoria sono di Google, ma non è l’unica azienda a ricorrere a bot per questa esigenza.

• Google
  • Mediapartners-Google
  • AdsBot-Google
  • AdsBot-Google-Mobile
  • AdsBot-Google-Mobile-Apps
• Microsoft
  • AdIdxBot

Oltre agli spider dei motori di ricerca e quelli per la corretta gestione degli annunci pubblicitari (utili solo per i siti che ne fanno uso), esistono diversi altri "bot buoni" come, ad esempio, quelli dei principali social network o di aziende come Apple.

Anche questi bot, infatti, possono essere utili al nostro sito. Grazie al loro passaggio, ad esempio, Facebook e Twitter sono grado di consentire la citazione dei nostri contenuti all’interno dei post degli utenti, Apple può suggerire le nostre pagine mediante l’assistente vocale SIRI o Spotlight, ecc.

Ancora una volta, quindi, i bot sono nostri amici in quanto in grado di portare una qualche utilità al nostro sito web.

• Apple
  • Applebot
• Facebook
  • facebookexternalhit
  • Facebot
• LinkedIn
  • LinkedInBot
• Pinterest
  • Pinterest
  • Pinterestbot
• Twitter
  • Twitterbot

Bot inutili e/o dannosi: questi sono i Bad Bots!

Non tutti i bot, tuttavia, sono buoni. Alcuni (anzi, la maggior parte) sono inutili per il nostro sito se non addirittura dannosi.

Esistono centinaia di bot "a spasso" per la Rete al fine di raccogliere informazioni per svariate finalità (ad esempio: indagini di mercato, raccolta di informazioni su specifici argomenti, cattura di immagini e video, ecc.). In tutti questi casi, solitamente, il bot non porta un beneficio diretto al sito web che visita ma ad altre persone.

Quando il bot è inutile o, peggio, pericoloso si utilizza l’espressione Bad Bot (robot cattivo) ad indicare che la sua visita non è gradita.

Secondo le statistiche circa il 20% del traffico web è generato dai Bad Bots: questo dovrebbe rendere l’idea delle dimensioni del fenomeno e di perché il "bravo webmaster" dovrebbe preoccuparsene.

Perché i Bad Bots sono pericolosi?

I Bad Bots sono pericolosi soprattutto perchè possono compromettere le performances di un sito web se non, addirittura, renderlo irraggiungibile per gli utenti umani ed i bot buoni!

Questo accade perchè i bot (sia quelli buoni che quelli cattivi), quando visitano un sito web ne consumano delle risorse e quando le visite sono parecchie c’è il rischio che il sito web ne risenta diventando lento o, peggio ancora, irraggiungibile.

E’ evidente che, in una situazione del genere, il passaggio frequente di Bad Bots diventa un problema da risolvere.

Come escludere i Bad Bots dal nostro sito?

Escludere i Bad Bots mediante robots.txt

Tutti i bot "legittimi", buoni o cattivi che siano, dovrebbero rispettare le direttive contenute nel file robots.txt, quindi possiamno pensare di escludere quelli sgraditi mediante l’indicazione del corrispondente User-Agent seguito dalla direttiva Disallow.

Di seguito il file robots.txt precompilato con un elenco molto corposo di bot che potrebbero consumare le preziose risorse del vostro sito web senza apportarvi alcuna utilità diretta:

# ESCLUDO LA SCANSIONE DEL SITO DA PARTE DI ALCUNI BOT

User-agent: 360Spider
Disallow:/
User-agent: 404checker
Disallow:/
User-agent: 404enemy
Disallow:/
User-agent: 80legs
Disallow:/
User-agent: Abonti
Disallow:/
User-agent: Aboundex
Disallow:/
User-agent: Aboundexbot
Disallow:/
User-agent: Acunetix
Disallow:/
User-agent: ADmantX
Disallow:/
User-agent: AfD-Verbotsverfahren
Disallow:/
User-agent: AhrefsBot
Disallow:/
User-agent: AIBOT
Disallow:/
User-agent: AiHitBot
Disallow:/
User-agent: Aipbot
Disallow:/
User-agent: Alexibot
Disallow:/
User-agent: Alligator
Disallow:/
User-agent: AllSubmitter
Disallow:/
User-agent: AlphaBot
Disallow:/
User-agent: Anarchie
Disallow:/
User-agent: Apexoo
Disallow:/
User-agent: archive.org_bot
Disallow:/
User-agent: ASPSeek
Disallow:/
User-agent: Asterias
Disallow:/
User-agent: Attach
Disallow:/
User-agent: autoemailspider
Disallow:/
User-agent: BackDoorBot
Disallow:/
User-agent: Backlink-Ceck
Disallow:/
User-agent: backlink-check
Disallow:/
User-agent: BacklinkCrawler
Disallow:/
User-agent: BackStreet
Disallow:/
User-agent: BackWeb
Disallow:/
User-agent: Badass
Disallow:/
User-agent: Bandit
Disallow:/
User-agent: Barkrowler
Disallow:/
User-agent: BatchFTP
Disallow:/
User-agent: Battleztar Bazinga
Disallow:/
User-agent: BBBike
Disallow:/
User-agent: BDCbot
Disallow:/
User-agent: BDFetch
Disallow:/
User-agent: BetaBot
Disallow:/
User-agent: Bigfoot
Disallow:/
User-agent: Bitacle
Disallow:/
User-agent: Blackboard
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlackWidow
Disallow:/
User-agent: BLEXBot
Disallow:/
User-agent: Blow
Disallow:/
User-agent: BlowFish
Disallow:/
User-agent: Boardreader
Disallow:/
User-agent: Bolt
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: Brandprotect
Disallow:/
User-agent: Brandwatch
Disallow:/
User-agent: Buddy
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: BuiltWith
Disallow:/
User-agent: Bullseye
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: BuzzSumo
Disallow:/
User-agent: Calculon
Disallow:/
User-agent: CATExplorador
Disallow:/
User-agent: CazoodleBot
Disallow:/
User-agent: CCBot
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CheTeam
Disallow:/
User-agent: ChinaClaw
Disallow:/
User-agent: Chlooe
Disallow:/
User-agent: Claritybot
Disallow:/
User-agent: Cliqzbot
Disallow:/
User-agent: Cloud mapping
Disallow:/
User-agent: coccocbot-web
Disallow:/
User-agent: Cogentbot
Disallow:/
User-agent: cognitiveseo
Disallow:/
User-agent: Collector
Disallow:/
User-agent: com.plumanalytics
Disallow:/
User-agent: Copier
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: Copyscape
Disallow:/
User-agent: Cosmos
Disallow:/
User-agent: Craftbot
Disallow:/
User-agent: crawler4j
Disallow:/
User-agent: crawler.feedback
Disallow:/
User-agent: crawl.sogou.com
Disallow:/
User-agent: CrazyWebCrawler
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: CrunchBot
Disallow:/
User-agent: CSHttp
Disallow:/
User-agent: Curious
Disallow:/
User-agent: Custo
Disallow:/
User-agent: DatabaseDriverMysqli
Disallow:/
User-agent: DataCha0s
Disallow:/
User-agent: DBLBot
Disallow:/
User-agent: demandbase-bot
Disallow:/
User-agent: Demon
Disallow:/
User-agent: Deusu
Disallow:/
User-agent: Devil
Disallow:/
User-agent: Digincore
Disallow:/
User-agent: DigitalPebble
Disallow:/
User-agent: DIIbot
Disallow:/
User-agent: Dirbuster
Disallow:/
User-agent: Disco
Disallow:/
User-agent: Discobot
Disallow:/
User-agent: Discoverybot
Disallow:/
User-agent: Dispatch
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: DnyzBot
Disallow:/
User-agent: DomainAppender
Disallow:/
User-agent: DomainCrawler
Disallow:/
User-agent: DomainSigmaCrawler
Disallow:/
User-agent: DomainStatsBot
Disallow:/
User-agent: Dotbot
Disallow:/
User-agent: Download Wonder
Disallow:/
User-agent: Dragonfly
Disallow:/
User-agent: Drip
Disallow:/
User-agent: DSearch
Disallow:/
User-agent: DTS Agent
Disallow:/
User-agent: EasyDL
Disallow:/
User-agent: Ebingbong
Disallow:/
User-agent: eCatch
Disallow:/
User-agent: ECCP/1.0
Disallow:/
User-agent: Ecxi
Disallow:/
User-agent: EirGrabber
Disallow:/
User-agent: EMail Siphon
Disallow:/
User-agent: EMail Wolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: evc-batch
Disallow:/
User-agent: Evil
Disallow:/
User-agent: Exabot
Disallow:/
User-agent: Express WebPictures
Disallow:/
User-agent: ExtLinksBot
Disallow:/
User-agent: Extractor
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Extreme Picture Finder
Disallow:/
User-agent: EyeNetIE
Disallow:/
User-agent: Ezooms
Disallow:/
User-agent: facebookscraper
Disallow:/
User-agent: FDM
Disallow:/
User-agent: FemtosearchBot
Disallow:/
User-agent: FHscan
Disallow:/
User-agent: Fimap
Disallow:/
User-agent: Firefox/7.0
Disallow:/
User-agent: FlashGet
Disallow:/
User-agent: Flunky
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Freeuploader
Disallow:/
User-agent: FrontPage
Disallow:/
User-agent: FyberSpider
Disallow:/
User-agent: Fyrebot
Disallow:/
User-agent: GalaxyBot
Disallow:/
User-agent: Genieo
Disallow:/
User-agent: GermCrawler
Disallow:/
User-agent: Getintent
Disallow:/
User-agent: GetRight
Disallow:/
User-agent: GetWeb
Disallow:/
User-agent: Gigablast
Disallow:/
User-agent: Gigabot
Disallow:/
User-agent: G-i-g-a-b-o-t
Disallow:/
User-agent: Go-Ahead-Got-It
Disallow:/
User-agent: Gotit
Disallow:/
User-agent: GoZilla
Disallow:/
User-agent: Go!Zilla
Disallow:/
User-agent: Grabber
Disallow:/
User-agent: GrabNet
Disallow:/
User-agent: Grafula
Disallow:/
User-agent: GrapeFX
Disallow:/
User-agent: GrapeshotCrawler
Disallow:/
User-agent: GridBot
Disallow:/
User-agent: GT::WWW
Disallow:/
User-agent: Haansoft
Disallow:/
User-agent: HaosouSpider
Disallow:/
User-agent: Harvest
Disallow:/
User-agent: Havij
Disallow:/
User-agent: HEADMasterSEO
Disallow:/
User-agent: Heritrix
Disallow:/
User-agent: Hloader
Disallow:/
User-agent: HMView
Disallow:/
User-agent: HTMLparser
Disallow:/
User-agent: HTTP::Lite
Disallow:/
User-agent: HTTrack
Disallow:/
User-agent: Humanlinks
Disallow:/
User-agent: HybridBot
Disallow:/
User-agent: Iblog
Disallow:/
User-agent: IDBot
Disallow:/
User-agent: Id-search
Disallow:/
User-agent: IlseBot
Disallow:/
User-agent: Image Fetch
Disallow:/
User-agent: Image Sucker
Disallow:/
User-agent: IndeedBot
Disallow:/
User-agent: Indy Library
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: InfoTekies
Disallow:/
User-agent: instabid
Disallow:/
User-agent: Intelliseek
Disallow:/
User-agent: InterGET
Disallow:/
User-agent: Internet Ninja
Disallow:/
User-agent: InternetSeer
Disallow:/
User-agent: internetVista monitor
Disallow:/
User-agent: ips-agent
Disallow:/
User-agent: Iria
Disallow:/
User-agent: IRLbot
Disallow:/
User-agent: Iskanie
Disallow:/
User-agent: IstellaBot
Disallow:/
User-agent: JamesBOT
Disallow:/
User-agent: Jbrofuzz
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: JetCar
Disallow:/
User-agent: Jetty
Disallow:/
User-agent: JikeSpider
Disallow:/
User-agent: JOC Web Spider
Disallow:/
User-agent: Joomla
Disallow:/
User-agent: Jorgee
Disallow:/
User-agent: JustView
Disallow:/
User-agent: Jyxobot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density
Disallow:/
User-agent: Kozmosbot
Disallow:/
User-agent: Lanshanbot
Disallow:/
User-agent: Larbin
Disallow:/
User-agent: LeechFTP
Disallow:/
User-agent: LeechGet
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: Lftp
Disallow:/
User-agent: LibWeb
Disallow:/
User-agent: Libwhisker
Disallow:/
User-agent: Lightspeedsystems
Disallow:/
User-agent: Likse
Disallow:/
User-agent: Linkdexbot
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkpadBot
Disallow:/
User-agent: LinkScan
Disallow:/
User-agent: LinksManager
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LinqiaMetadataDownloaderBot
Disallow:/
User-agent: LinqiaRSSBot
Disallow:/
User-agent: LinqiaScrapeBot
Disallow:/
User-agent: Lipperhey
Disallow:/
User-agent: Lipperhey Spider
Disallow:/
User-agent: Litemage_walker
Disallow:/
User-agent: Lmspider
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: Ltx71
Disallow:/
User-agent: lwp-request
Disallow:/
User-agent: LWP::Simple
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: Magnet
Disallow:/
User-agent: Mag-Net
Disallow:/
User-agent: magpie-crawler
Disallow:/
User-agent: Mail.RU_Bot
Disallow:/
User-agent: Majestic12
Disallow:/
User-agent: Majestic-SEO
Disallow:/
User-agent: Majestic SEO
Disallow:/
User-agent: MarkMonitor
Disallow:/
User-agent: MarkWatch
Disallow:/
User-agent: Masscan
Disallow:/
User-agent: Mass Downloader
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: MauiBot
Disallow:/
User-agent: meanpathbot
Disallow:/
User-agent: Meanpathbot
Disallow:/
User-agent: MeanPath Bot
Disallow:/
User-agent: Mediatoolkitbot
Disallow:/
User-agent: mediawords
Disallow:/
User-agent: MegaIndex.ru
Disallow:/
User-agent: Metauri
Disallow:/
User-agent: MFC_Tear_Sample
Disallow:/
User-agent: Microsoft Data Access
Disallow:/
User-agent: Microsoft URL Control
Disallow:/
User-agent: MIDown tool
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: MJ12bot
Disallow:/
User-agent: Mojeek
Disallow:/
User-agent: Morfeus Fucking Scanner
Disallow:/
User-agent: Mr.4x3
Disallow:/
User-agent: MSFrontPage
Disallow:/
User-agent: MSIECrawler
Disallow:/
User-agent: Msrabot
Disallow:/
User-agent: MS Web Services Client Protocol
Disallow:/
User-agent: muhstik-scan
Disallow:/
User-agent: Musobot
Disallow:/
User-agent: Name Intelligence
Disallow:/
User-agent: Nameprotect
Disallow:/
User-agent: Navroad
Disallow:/
User-agent: NearSite
Disallow:/
User-agent: Needle
Disallow:/
User-agent: Nessus
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: Netcraft
Disallow:/
User-agent: netEstate NE Crawler
Disallow:/
User-agent: NetLyzer
Disallow:/
User-agent: NetMechanic
Disallow:/
User-agent: NetSpider
Disallow:/
User-agent: Nettrack
Disallow:/
User-agent: Net Vampire
Disallow:/
User-agent: Netvibes
Disallow:/
User-agent: NetZIP
Disallow:/
User-agent: NextGenSearchBot
Disallow:/
User-agent: Nibbler
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Niki-bot
Disallow:/
User-agent: Nikto
Disallow:/
User-agent: NimbleCrawler
Disallow:/
User-agent: Nimbostratus
Disallow:/
User-agent: Ninja
Disallow:/
User-agent: Nmap
Disallow:/
User-agent: NPbot
Disallow:/
User-agent: Nutch
Disallow:/
User-agent: oBot
Disallow:/
User-agent: Octopus
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Offline Navigator
Disallow:/
User-agent: OnCrawl
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: OpenLinkProfiler
Disallow:/
User-agent: Openvas
Disallow:/
User-agent: OpenVAS
Disallow:/
User-agent: OrangeBot
Disallow:/
User-agent: OrangeSpider
Disallow:/
User-agent: OutclicksBot
Disallow:/
User-agent: OutfoxBot
Disallow:/
User-agent: PageAnalyzer
Disallow:/
User-agent: Page Analyzer
Disallow:/
User-agent: PageGrabber
Disallow:/
User-agent: page scorer
Disallow:/
User-agent: PageScorer
Disallow:/
User-agent: Pandalytics
Disallow:/
User-agent: Panscient
Disallow:/
User-agent: Papa Foto
Disallow:/
User-agent: Pavuk
Disallow:/
User-agent: pcBrowser
Disallow:/
User-agent: PECL::HTTP
Disallow:/
User-agent: PeoplePal
Disallow:/
User-agent: PHPCrawl
Disallow:/
User-agent: Picscout
Disallow:/
User-agent: Picsearch
Disallow:/
User-agent: PictureFinder
Disallow:/
User-agent: Pimonster
Disallow:/
User-agent: Pi-Monster
Disallow:/
User-agent: Pixray
Disallow:/
User-agent: PleaseCrawl
Disallow:/
User-agent: plumanalytics
Disallow:/
User-agent: Pockey
Disallow:/
User-agent: POE-Component-Client-HTTP
Disallow:/
User-agent: Probethenet
Disallow:/
User-agent: ProPowerBot
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: Psbot
Disallow:/
User-agent: Pump
Disallow:/
User-agent: PxBroker
Disallow:/
User-agent: PyCurl
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: Quick-Crawler
Disallow:/
User-agent: RankActive
Disallow:/
User-agent: RankActiveLinkBot
Disallow:/
User-agent: RankFlex
Disallow:/
User-agent: RankingBot
Disallow:/
User-agent: RankingBot2
Disallow:/
User-agent: Rankivabot
Disallow:/
User-agent: RankurBot
Disallow:/
User-agent: RealDownload
Disallow:/
User-agent: Reaper
Disallow:/
User-agent: RebelMouse
Disallow:/
User-agent: Recorder
Disallow:/
User-agent: RedesScrapy
Disallow:/
User-agent: ReGet
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: Ripper
Disallow:/
User-agent: RocketCrawler
Disallow:/
User-agent: Rogerbot
Disallow:/
User-agent: s1z.ru
Disallow:/
User-agent: SalesIntelligent
Disallow:/
User-agent: SBIder
Disallow:/
User-agent: ScanAlert
Disallow:/
User-agent: Scanbot
Disallow:/
User-agent: scan.lol
Disallow:/
User-agent: ScoutJet
Disallow:/
User-agent: Scrapy
Disallow:/
User-agent: Screaming
Disallow:/
User-agent: ScreenerBot
Disallow:/
User-agent: Searchestate
Disallow:/
User-agent: SearchmetricsBot
Disallow:/
User-agent: Semrush
Disallow:/
User-agent: SemrushBot
Disallow:/
User-agent: SEOkicks
Disallow:/
User-agent: SEOkicks-Robot
Disallow:/
User-agent: SEOlyticsCrawler
Disallow:/
User-agent: Seomoz
Disallow:/
User-agent: SEOprofiler
Disallow:/
User-agent: seoscanners
Disallow:/
User-agent: SeoSiteCheckup
Disallow:/
User-agent: SEOstats
Disallow:/
User-agent: serpstatbot
Disallow:/
User-agent: sexsearcher
Disallow:/
User-agent: Shodan
Disallow:/
User-agent: Siphon
Disallow:/
User-agent: SISTRIX
Disallow:/
User-agent: Sitebeam
Disallow:/
User-agent: SiteExplorer
Disallow:/
User-agent: Siteimprove
Disallow:/
User-agent: SiteLockSpider
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SiteSucker
Disallow:/
User-agent: Site Sucker
Disallow:/
User-agent: Sitevigil
Disallow:/
User-agent: SlySearch
Disallow:/
User-agent: SmartDownload
Disallow:/
User-agent: SMTBot
Disallow:/
User-agent: Snake
Disallow:/
User-agent: Snapbot
Disallow:/
User-agent: Snoopy
Disallow:/
User-agent: SocialRankIOBot
Disallow:/
User-agent: Sociscraper
Disallow:/
User-agent: sogouspider
Disallow:/
User-agent: Sogou web spider
Disallow:/
User-agent: Sosospider
Disallow:/
User-agent: Sottopop
Disallow:/
User-agent: SpaceBison
Disallow:/
User-agent: Spammen
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: Spanner
Disallow:/
User-agent: sp_auditbot
Disallow:/
User-agent: Spbot
Disallow:/
User-agent: Spinn3r
Disallow:/
User-agent: SputnikBot
Disallow:/
User-agent: spyfu
Disallow:/
User-agent: Sqlmap
Disallow:/
User-agent: Sqlworm
Disallow:/
User-agent: Sqworm
Disallow:/
User-agent: Steeler
Disallow:/
User-agent: Stripper
Disallow:/
User-agent: Sucker
Disallow:/
User-agent: Sucuri
Disallow:/
User-agent: SuperBot
Disallow:/
User-agent: SuperHTTP
Disallow:/
User-agent: Surfbot
Disallow:/
User-agent: SurveyBot
Disallow:/
User-agent: Suzuran
Disallow:/
User-agent: Swiftbot
Disallow:/
User-agent: sysscan
Disallow:/
User-agent: Szukacz
Disallow:/
User-agent: T0PHackTeam
Disallow:/
User-agent: T8Abot
Disallow:/
User-agent: tAkeOut
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: Telesphoreo
Disallow:/
User-agent: Telesphorep
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: Thumbor
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: Toata
Disallow:/
User-agent: Toweyabot
Disallow:/
User-agent: Tracemyfile
Disallow:/
User-agent: Trendiction
Disallow:/
User-agent: Trendictionbot
Disallow:/
User-agent: trendiction.com
Disallow:/
User-agent: trendiction.de
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: Turingos
Disallow:/
User-agent: Turnitin
Disallow:/
User-agent: TurnitinBot
Disallow:/
User-agent: TwengaBot
Disallow:/
User-agent: Twice
Disallow:/
User-agent: Typhoeus
Disallow:/
User-agent: UnisterBot
Disallow:/
User-agent: Upflow
Disallow:/
User-agent: URLy.Warning
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: Vacuum
Disallow:/
User-agent: Vagabondo
Disallow:/
User-agent: VB Project
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VeriCiteCrawler
Disallow:/
User-agent: VidibleScraper
Disallow:/
User-agent: Virusdie
Disallow:/
User-agent: VoidEYE
Disallow:/
User-agent: Voil
Disallow:/
User-agent: Voltron
Disallow:/
User-agent: Wallpapers/3.0
Disallow:/
User-agent: WallpapersHD
Disallow:/
User-agent: WASALive-Bot
Disallow:/
User-agent: WBSearchBot
Disallow:/
User-agent: Webalta
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: Web Auto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebCollage
Disallow:/
User-agent: Web Collage
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WEBDAV
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: Web Enhancer
Disallow:/
User-agent: WebFetch
Disallow:/
User-agent: Web Fetch
Disallow:/
User-agent: WebFuck
Disallow:/
User-agent: Web Fuck
Disallow:/
User-agent: WebGo IS
Disallow:/
User-agent: WebImageCollector
Disallow:/
User-agent: WebLeacher
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: webmeup-crawler
Disallow:/
User-agent: WebPix
Disallow:/
User-agent: Web Pix
Disallow:/
User-agent: WebReaper
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Web Sauger
Disallow:/
User-agent: Webshag
Disallow:/
User-agent: WebsiteExtractor
Disallow:/
User-agent: WebsiteQuester
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebSucker
Disallow:/
User-agent: Web Sucker
Disallow:/
User-agent: WebWhacker
Disallow:/
User-agent: WebZIP
Disallow:/
User-agent: WeSEE
Disallow:/
User-agent: Whack
Disallow:/
User-agent: Whacker
Disallow:/
User-agent: Whatweb
Disallow:/
User-agent: Who.is Bot
Disallow:/
User-agent: Widow
Disallow:/
User-agent: WinHTTrack
Disallow:/
User-agent: WiseGuys Robot
Disallow:/
User-agent: WISENutbot
Disallow:/
User-agent: Wonderbot
Disallow:/
User-agent: Woobot
Disallow:/
User-agent: Wotbox
Disallow:/
User-agent: Wprecon
Disallow:/
User-agent: WPScan
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: WWW-Mechanize
Disallow:/
User-agent: WWW::Mechanize
Disallow:/
User-agent: WWWOFFLE
Disallow:/
User-agent: x09Mozilla
Disallow:/
User-agent: x22Mozilla
Disallow:/
User-agent: Xaldon_WebSpider
Disallow:/
User-agent: Xaldon WebSpider
Disallow:/
User-agent: Xenu
Disallow:/
User-agent: xpymep1.exe
Disallow:/
User-agent: YoudaoBot
Disallow:/
User-agent: Zade
Disallow:/
User-agent: Zauba
Disallow:/
User-agent: zauba.io
Disallow:/
User-agent: Zermelo
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: zgrab
Disallow:/
User-agent: Zitebot
Disallow:/
User-agent: ZmEu
Disallow:/
User-agent: ZumBot
Disallow:/
User-agent: ZyBorg
Disallow:/

Escludere i Bad Bots mediante .htaccess

Purtroppo non tutti i Bad Bots rispettano le regole contenute nel file robots.txt. I bot più aggressivi e pericolosi, ovviamente, non sono interessati alle direttive del webmaster del sito e, pertanto, tendono ad ignorale.

In questi casi, quindi, il file robots.txt non si rivela utile e bisogna intervenire in altri modi, ad esempio mediante il file .htaccess.

Per farlo possiamo, ancora una volta, fare affidamento su regole basate sullo User-Agent. Ecco un esempio di codice per filtrare i bad Bots con .htaccess in base al riconoscimento dello user agent:

Di seguito il blocco da copiare e incollare nel vostro file .htaccess.

# Blocco i Bad Bots
<IfModule mod_setenvif.c>
# SetEnvIfNoCase User-Agent ^$ bad_bot
SetEnvIfNoCase User-Agent "^12soso.*" bad_bot
SetEnvIfNoCase User-Agent "^192.comAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^1Noonbot.*" bad_bot
SetEnvIfNoCase User-Agent "^1on1searchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^3D_SEARCH.*" bad_bot
SetEnvIfNoCase User-Agent "^3DE_SEARCH2.*" bad_bot
SetEnvIfNoCase User-Agent "^3GSE.*" bad_bot
SetEnvIfNoCase User-Agent "^50.nu.*" bad_bot
SetEnvIfNoCase User-Agent "^A1 Sitemap Generator.*" bad_bot
SetEnvIfNoCase User-Agent "^A1 Website Download.*" bad_bot
SetEnvIfNoCase User-Agent "^A6-Indexer.*" bad_bot
SetEnvIfNoCase User-Agent "^AASP.*" bad_bot
SetEnvIfNoCase User-Agent "^ABACHOBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Abonti.*" bad_bot
SetEnvIfNoCase User-Agent "^abot.*" bad_bot
SetEnvIfNoCase User-Agent "^AbotEmailSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^AboutUsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AccMonitor Compliance Server.*" bad_bot
SetEnvIfNoCase User-Agent "^accoona.*" bad_bot
SetEnvIfNoCase User-Agent "^AChulkov.NET page walker.*" bad_bot
SetEnvIfNoCase User-Agent "^Acme.Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^AcoonBot.*" bad_bot
SetEnvIfNoCase User-Agent "^acquia-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^ActiveTouristBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ad Muncher.*" bad_bot
SetEnvIfNoCase User-Agent "^AdamM Bot,.*" bad_bot
SetEnvIfNoCase User-Agent "^adbeat_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^AdMedia bot.*" bad_bot
SetEnvIfNoCase User-Agent "^adminshop.com.*" bad_bot
SetEnvIfNoCase User-Agent "^AdsBot-Google.*" bad_bot
SetEnvIfNoCase User-Agent "^Advanced Email Extractor.*" bad_bot
SetEnvIfNoCase User-Agent "^AESOP_com_SpiderMan.*" bad_bot
SetEnvIfNoCase User-Agent "^AESpider.*" bad_bot
SetEnvIfNoCase User-Agent "^AF Knowledge Now Verity Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^aggregator:Vocus.*" bad_bot
SetEnvIfNoCase User-Agent "^ah-ha.com crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^AhrefsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AhrefsBot/5.0.*" bad_bot
SetEnvIfNoCase User-Agent "^AIBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^aiHitBot.*" bad_bot
SetEnvIfNoCase User-Agent "^aipbot.*" bad_bot
SetEnvIfNoCase User-Agent "^AISIID.*" bad_bot
SetEnvIfNoCase User-Agent "^AITCSRobot/1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Akamai-SiteSnapshot.*" bad_bot
SetEnvIfNoCase User-Agent "^AlexaWebSearchPlatform.*" bad_bot
SetEnvIfNoCase User-Agent "^AlexfDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^Alexibot.*" bad_bot
SetEnvIfNoCase User-Agent "^AlkalineBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^All Acronyms Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Alpha Search Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^Amerla Search Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Amfibibot.*" bad_bot
SetEnvIfNoCase User-Agent "^AmPmPPC.com.*" bad_bot
SetEnvIfNoCase User-Agent "^AmPmPPC.com.*" bad_bot
SetEnvIfNoCase User-Agent "^AMZNKAssocBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Anemone.*" bad_bot
SetEnvIfNoCase User-Agent "^Anonymous/3G bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Anonymouse.org.*" bad_bot
SetEnvIfNoCase User-Agent "^AnotherBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AnswerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AnswerBus.*" bad_bot
SetEnvIfNoCase User-Agent "^AnswerChase PROve.*" bad_bot
SetEnvIfNoCase User-Agent "^AntBot.*" bad_bot
SetEnvIfNoCase User-Agent "^antibot-.*" bad_bot
SetEnvIfNoCase User-Agent "^AntiSantyWorm.*" bad_bot
SetEnvIfNoCase User-Agent "^Antro.Net.*" bad_bot
SetEnvIfNoCase User-Agent "^AONDE-Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Apache-HttpClient.*" bad_bot
SetEnvIfNoCase User-Agent "^Apache-HttpClient/4.0.1 (java 1.5).*" bad_bot
SetEnvIfNoCase User-Agent "^ApacheBench.*" bad_bot
SetEnvIfNoCase User-Agent "^Aport.*" bad_bot
SetEnvIfNoCase User-Agent "^appid: s~stremor-crawler-.*" bad_bot
SetEnvIfNoCase User-Agent "^Aqua_Products.*" bad_bot
SetEnvIfNoCase User-Agent "^AraBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Arachmo.*" bad_bot
SetEnvIfNoCase User-Agent "^Arachnophilia.*" bad_bot
SetEnvIfNoCase User-Agent "^archive.org_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^archive.org_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^aria eQualizer.*" bad_bot
SetEnvIfNoCase User-Agent "^arianna.libero.it.*" bad_bot
SetEnvIfNoCase User-Agent "^Arikus_Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Art-Online.com.*" bad_bot
SetEnvIfNoCase User-Agent "^ArtavisBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Artera.*" bad_bot
SetEnvIfNoCase User-Agent "^ASAHA Search Engine Turkey.*" bad_bot
SetEnvIfNoCase User-Agent "^ASpider/0.09.*" bad_bot
SetEnvIfNoCase User-Agent "^ASPSeek.*" bad_bot
SetEnvIfNoCase User-Agent "^ASPseek.*" bad_bot
SetEnvIfNoCase User-Agent "^asterias.*" bad_bot
SetEnvIfNoCase User-Agent "^AstroFind.*" bad_bot
SetEnvIfNoCase User-Agent "^athenusbot.*" bad_bot
SetEnvIfNoCase User-Agent "^AtlocalBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Atomic_Email_Hunter.*" bad_bot
SetEnvIfNoCase User-Agent "^attach.*" bad_bot
SetEnvIfNoCase User-Agent "^attrakt.*" bad_bot
SetEnvIfNoCase User-Agent "^Attributor.comBot.*" bad_bot
SetEnvIfNoCase User-Agent "^attributor.*" bad_bot
SetEnvIfNoCase User-Agent "^Attributor/Dejan-.*" bad_bot
SetEnvIfNoCase User-Agent "^augurfind.*" bad_bot
SetEnvIfNoCase User-Agent "^AURESYS.*" bad_bot
SetEnvIfNoCase User-Agent "^AutoBaron crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^autoemailspider.*" bad_bot
SetEnvIfNoCase User-Agent "^autowebdir.*" bad_bot
SetEnvIfNoCase User-Agent "^AVSearch-.*" bad_bot
SetEnvIfNoCase User-Agent "^axfeedsbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Axonize-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ayna.*" bad_bot
SetEnvIfNoCase User-Agent "^b2w.*" bad_bot
SetEnvIfNoCase User-Agent "^b2w/0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BackRub/..*" bad_bot
SetEnvIfNoCase User-Agent "^BackStreet Browser.*" bad_bot
SetEnvIfNoCase User-Agent "^BackWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider.*" bad_bot
SetEnvIfNoCase User-Agent "^BaiDuSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^BaiduSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider-video.*" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider+.*" bad_bot
SetEnvIfNoCase User-Agent "^Bandit.*" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP.*" bad_bot
SetEnvIfNoCase User-Agent "^baypup.*" bad_bot
SetEnvIfNoCase User-Agent "^BDFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^BecomeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BecomeJPBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BeetleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Bender.*" bad_bot
SetEnvIfNoCase User-Agent "^Bender.*" bad_bot
SetEnvIfNoCase User-Agent "^besserscheitern-crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^betaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Big Brother.*" bad_bot
SetEnvIfNoCase User-Agent "^Big Brother.*" bad_bot
SetEnvIfNoCase User-Agent "^Big Data.*" bad_bot
SetEnvIfNoCase User-Agent "^Bigado.com.*" bad_bot
SetEnvIfNoCase User-Agent "^BigCliqueBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BigCliqueBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot.*" bad_bot
SetEnvIfNoCase User-Agent "^BIGLOTRON.*" bad_bot
SetEnvIfNoCase User-Agent "^Bilbo.*" bad_bot
SetEnvIfNoCase User-Agent "^BilgiBetaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BilgiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^bintellibot.*" bad_bot
SetEnvIfNoCase User-Agent "^bitlybot.*" bad_bot
SetEnvIfNoCase User-Agent "^BitvoUserAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Bizbot003.*" bad_bot
SetEnvIfNoCase User-Agent "^BizBot04 kirk.overleaf.com.*" bad_bot
SetEnvIfNoCase User-Agent "^BizWorks Retriever.*" bad_bot
SetEnvIfNoCase User-Agent "^Black Hole.*" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole.*" bad_bot
SetEnvIfNoCase User-Agent "^Blackbird.*" bad_bot
SetEnvIfNoCase User-Agent "^BlackMask.Net Search Engine.*" bad_bot
SetEnvIfNoCase User-Agent "^BlackWidow.*" bad_bot
SetEnvIfNoCase User-Agent "^bladder fusion.*" bad_bot
SetEnvIfNoCase User-Agent "^Blaiz-Bee.*" bad_bot
SetEnvIfNoCase User-Agent "^BLEXBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BLEXBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Blinkx/DFS-Fetch.*" bad_bot
SetEnvIfNoCase User-Agent "^BlitzBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Blog Conversation Project.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogMyWay.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogPulseLive.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogRefsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogScope.*" bad_bot
SetEnvIfNoCase User-Agent "^Blogslive.*" bad_bot
SetEnvIfNoCase User-Agent "^BloobyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish.*" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish.*" bad_bot
SetEnvIfNoCase User-Agent "^BLT.*" bad_bot
SetEnvIfNoCase User-Agent "^bnf.fr_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^BoaConstrictor.*" bad_bot
SetEnvIfNoCase User-Agent "^BoardReader-Image-Fetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^BOI_crawl_00.*" bad_bot
SetEnvIfNoCase User-Agent "^BOIA-Scan-Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^BOIA.ORG-Scan-Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^boitho.com-dc.*" bad_bot
SetEnvIfNoCase User-Agent "^Bookmark Buddy bookmark checker.*" bad_bot
SetEnvIfNoCase User-Agent "^Bookmark search tool.*" bad_bot
SetEnvIfNoCase User-Agent "^bosug.*" bad_bot
SetEnvIfNoCase User-Agent "^Bot Apoena.*" bad_bot
SetEnvIfNoCase User-Agent "^Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^bot.*" bad_bot
SetEnvIfNoCase User-Agent "^BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^BotALot.*" bad_bot
SetEnvIfNoCase User-Agent "^BotRightHere.*" bad_bot
SetEnvIfNoCase User-Agent "^Botswana.*" bad_bot
SetEnvIfNoCase User-Agent "^bottybot.*" bad_bot
SetEnvIfNoCase User-Agent "^BpBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BRAINTIME_SEARCH.*" bad_bot
SetEnvIfNoCase User-Agent "^BrokenLinkCheck.com.*" bad_bot
SetEnvIfNoCase User-Agent "^BrowserEmulator.*" bad_bot
SetEnvIfNoCase User-Agent "^BrowserMob.*" bad_bot
SetEnvIfNoCase User-Agent "^BruinBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BSearchR&D.*" bad_bot
SetEnvIfNoCase User-Agent "^BSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^btbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Btsearch.*" bad_bot
SetEnvIfNoCase User-Agent "^Buddy.*" bad_bot
SetEnvIfNoCase User-Agent "^Buibui.*" bad_bot
SetEnvIfNoCase User-Agent "^BuildCMS crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough.*" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye.*" bad_bot
SetEnvIfNoCase User-Agent "^bumblebee.*" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers.*" bad_bot
SetEnvIfNoCase User-Agent "^BuscadorClarin.*" bad_bot
SetEnvIfNoCase User-Agent "^Buscaplus Robi.*" bad_bot
SetEnvIfNoCase User-Agent "^Butterfly.*" bad_bot
SetEnvIfNoCase User-Agent "^BuyHawaiiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BuzzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^byindia.*" bad_bot
SetEnvIfNoCase User-Agent "^BYINDIA.*" bad_bot
SetEnvIfNoCase User-Agent "^BySpider.*" bad_bot
SetEnvIfNoCase User-Agent "^byteserver.*" bad_bot
SetEnvIfNoCase User-Agent "^bzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^c r a w l 3 r.*" bad_bot
SetEnvIfNoCase User-Agent "^CacheBlaster.*" bad_bot
SetEnvIfNoCase User-Agent "^Caddbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Cafi.*" bad_bot
SetEnvIfNoCase User-Agent "^Camcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^CamelStampede.*" bad_bot
SetEnvIfNoCase User-Agent "^Canon-WebRecord.*" bad_bot
SetEnvIfNoCase User-Agent "^Canon-WebRecordPro.*" bad_bot
SetEnvIfNoCase User-Agent "^CareerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^cataguru.*" bad_bot
SetEnvIfNoCase User-Agent "^CatchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CatchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CazoodleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CCBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CCGCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^ccubee.*" bad_bot
SetEnvIfNoCase User-Agent "^CD-Preload.*" bad_bot
SetEnvIfNoCase User-Agent "^CE-Preload.*" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh.*" bad_bot
SetEnvIfNoCase User-Agent "^Cerberian Drtrs.*" bad_bot
SetEnvIfNoCase User-Agent "^CERT FigleafBot.*" bad_bot
SetEnvIfNoCase User-Agent "^cfetch.*" bad_bot
SetEnvIfNoCase User-Agent "^CFNetwork.*" bad_bot
SetEnvIfNoCase User-Agent "^Chameleon.*" bad_bot
SetEnvIfNoCase User-Agent "^Charlotte.*" bad_bot
SetEnvIfNoCase User-Agent "^Check&Get.*" bad_bot
SetEnvIfNoCase User-Agent "^Checkbot.*" bad_bot
SetEnvIfNoCase User-Agent "^CheckLinks.*" bad_bot
SetEnvIfNoCase User-Agent "^Checklinks.*" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ChemieDE-NodeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker.*" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerElite.*" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerSE.*" bad_bot
SetEnvIfNoCase User-Agent "^Chilkat.*" bad_bot
SetEnvIfNoCase User-Agent "^ChinaClaw.*" bad_bot
SetEnvIfNoCase User-Agent "^CipinetBot.*" bad_bot
SetEnvIfNoCase User-Agent "^cis455crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^citeseerxbot.*" bad_bot
SetEnvIfNoCase User-Agent "^cizilla.com/Cizilla-.*" bad_bot
SetEnvIfNoCase User-Agent "^ClariaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Climate Ark.*" bad_bot
SetEnvIfNoCase User-Agent "^ClimateArk Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Clushbot.*" bad_bot
SetEnvIfNoCase User-Agent "^COAST scan engine.*" bad_bot
SetEnvIfNoCase User-Agent "^COAST WebMaster Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^coccoc.*" bad_bot
SetEnvIfNoCase User-Agent "^CollapsarWEB.*" bad_bot
SetEnvIfNoCase User-Agent "^Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^Combine.*" bad_bot
SetEnvIfNoCase User-Agent "^combine.*" bad_bot
SetEnvIfNoCase User-Agent "^COMBINE.*" bad_bot
SetEnvIfNoCase User-Agent "^Compatible.*" bad_bot
SetEnvIfNoCase User-Agent "^ConnectSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^conpilot.*" bad_bot
SetEnvIfNoCase User-Agent "^ContentSmartz.*" bad_bot
SetEnvIfNoCase User-Agent "^ContextAd Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^contype.*" bad_bot
SetEnvIfNoCase User-Agent "^contype.*" bad_bot
SetEnvIfNoCase User-Agent "^cookieNET.*" bad_bot
SetEnvIfNoCase User-Agent "^CoolBot .*" bad_bot
SetEnvIfNoCase User-Agent "^CoolCheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Copernic.*" bad_bot
SetEnvIfNoCase User-Agent "^Copier.*" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck.*" bad_bot
SetEnvIfNoCase User-Agent "^core-project.*" bad_bot
SetEnvIfNoCase User-Agent "^cosmos.*" bad_bot
SetEnvIfNoCase User-Agent "^cosmos.*" bad_bot
SetEnvIfNoCase User-Agent "^Covario-IDS.*" bad_bot
SetEnvIfNoCase User-Agent "^Cowbot-.*" bad_bot
SetEnvIfNoCase User-Agent "^Cowdog Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^crabbyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawl_Application.*" bad_bot
SetEnvIfNoCase User-Agent "^crawl.UserAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^CrawlConvera.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler_for_infomine.*" bad_bot
SetEnvIfNoCase User-Agent "^CRAWLER-ALTSE.VUNET.ORG-Lynx.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler-upgrade-config.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler.kpricorn.org.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler@.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler43.ejupiter.com.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler4j.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawly.*" bad_bot
SetEnvIfNoCase User-Agent "^CreativeCommons.*" bad_bot
SetEnvIfNoCase User-Agent "^Crescent.*" bad_bot
SetEnvIfNoCase User-Agent "^Crescent Internet ToolPak HTTP OLE Control v.1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^cs-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^CSE HTML Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^CSHttpClient.*" bad_bot
SetEnvIfNoCase User-Agent "^Cuasarbot.*" bad_bot
SetEnvIfNoCase User-Agent "^culsearch/culs.*" bad_bot
SetEnvIfNoCase User-Agent "^Curl.*" bad_bot
SetEnvIfNoCase User-Agent "^Custo.*" bad_bot
SetEnvIfNoCase User-Agent "^Custo.*" bad_bot
SetEnvIfNoCase User-Agent "^cvaulev.*" bad_bot
SetEnvIfNoCase User-Agent "^Cyberdog.*" bad_bot
SetEnvIfNoCase User-Agent "^CyberNavi_WebGet.*" bad_bot
SetEnvIfNoCase User-Agent "^CyberPatrol SiteCat Webbot.*" bad_bot
SetEnvIfNoCase User-Agent "^CyberSpyder.*" bad_bot
SetEnvIfNoCase User-Agent "^CydralSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^D1GArabicEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^DA.*" bad_bot
SetEnvIfNoCase User-Agent "^DataCha0s.*" bad_bot
SetEnvIfNoCase User-Agent "^DataFountains.*" bad_bot
SetEnvIfNoCase User-Agent "^DataFountains/DMOZ Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^DataparkSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^datascape robot.*" bad_bot
SetEnvIfNoCase User-Agent "^DataSpearSpiderBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DataSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Dattatec.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Dattatec.com-Sitios-Top.*" bad_bot
SetEnvIfNoCase User-Agent "^Daumoa.*" bad_bot
SetEnvIfNoCase User-Agent "^DAUMOA-video.*" bad_bot
SetEnvIfNoCase User-Agent "^DAUMOA-web.*" bad_bot
SetEnvIfNoCase User-Agent "^Daumoa.*" bad_bot
SetEnvIfNoCase User-Agent "^Declumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Deepindex.*" bad_bot
SetEnvIfNoCase User-Agent "^deepnet crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^DeepTrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^dejan.*" bad_bot
SetEnvIfNoCase User-Agent "^del.icio.us-thumbnails.*" bad_bot
SetEnvIfNoCase User-Agent "^DelvuBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Der große BilderSauger.*" bad_bot
SetEnvIfNoCase User-Agent "^DiaGem.*" bad_bot
SetEnvIfNoCase User-Agent "^Diamond.*" bad_bot
SetEnvIfNoCase User-Agent "^DiamondBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^didaxusbot.*" bad_bot
SetEnvIfNoCase User-Agent "^DigExt.*" bad_bot
SetEnvIfNoCase User-Agent "^Digger.*" bad_bot
SetEnvIfNoCase User-Agent "^DiGi-RSSBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DigitalArchivesBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DigOut4U.*" bad_bot
SetEnvIfNoCase User-Agent "^DIIbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dillo.*" bad_bot
SetEnvIfNoCase User-Agent "^Dir_Snatch.exe.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo Pump.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo Pump 3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo Pump 3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCoFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCoFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^Distilled-Reputation-Monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^Dit.*" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder.*" bad_bot
SetEnvIfNoCase User-Agent "^DjangoTraineeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DKIMRepBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DoCoMo.*" bad_bot
SetEnvIfNoCase User-Agent "^DoCoMo.*" bad_bot
SetEnvIfNoCase User-Agent "^DOF-Verify.*" bad_bot
SetEnvIfNoCase User-Agent "^Domaincrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^domaincrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^DomainScan.*" bad_bot
SetEnvIfNoCase User-Agent "^DomainWatcher Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^DotBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DotBot/1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^DotSpotsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dow Jones Searchbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Download.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Demon.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Demon/3.2.0.8.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Demon/3.5.0.11.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Devil.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Master.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Ninja.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Wonder.*" bad_bot
SetEnvIfNoCase User-Agent "^Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^DOY.*" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly.*" bad_bot
SetEnvIfNoCase User-Agent "^Drip.*" bad_bot
SetEnvIfNoCase User-Agent "^drone.*" bad_bot
SetEnvIfNoCase User-Agent "^Drupal.*" bad_bot
SetEnvIfNoCase User-Agent "^DTAAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^dtSearchSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^DuckDuckBot.*" bad_bot
SetEnvIfNoCase User-Agent "^dumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dwaar.*" bad_bot
SetEnvIfNoCase User-Agent "^Dwaarbot.*" bad_bot
SetEnvIfNoCase User-Agent "^DXSeeker.*" bad_bot
SetEnvIfNoCase User-Agent "^EAH.*" bad_bot
SetEnvIfNoCase User-Agent "^Earth Platform Indexer.*" bad_bot
SetEnvIfNoCase User-Agent "^Earth Science Educator  robot.*" bad_bot
SetEnvIfNoCase User-Agent "^EARTHCOM.info.*" bad_bot
SetEnvIfNoCase User-Agent "^EARTHCOM.*" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL.*" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL.*" bad_bot
SetEnvIfNoCase User-Agent "^ebingbong.*" bad_bot
SetEnvIfNoCase User-Agent "^EC2LinkFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^eCairn-Grabber.*" bad_bot
SetEnvIfNoCase User-Agent "^eCatch.*" bad_bot
SetEnvIfNoCase User-Agent "^eCatch.*" bad_bot
SetEnvIfNoCase User-Agent "^eCatch/3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^eChooseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EdisterBot (http://www.edister.com/bot.html)" bad_bot
SetEnvIfNoCase User-Agent "^EduGovSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^egothor.*" bad_bot
SetEnvIfNoCase User-Agent "^eidetica.com/spider.*" bad_bot
SetEnvIfNoCase User-Agent "^EirGrabber.*" bad_bot
SetEnvIfNoCase User-Agent "^Elblindo the Blind Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^ElisaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EllerdaleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EMail Exractor.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailLeach.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf.*" bad_bot
SetEnvIfNoCase User-Agent "^EMPAS_ROBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^EnaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^endeca.*" bad_bot
SetEnvIfNoCase User-Agent "^EnigmaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Enswer Neuro Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^EntityCubeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^es.*" bad_bot
SetEnvIfNoCase User-Agent "^eStyleSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^eSyndiCat Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Eurosoft-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Evaal.*" bad_bot
SetEnvIfNoCase User-Agent "^Eventware.*" bad_bot
SetEnvIfNoCase User-Agent "^Everest-Vulcan Inc..*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot Test.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot-Images.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot-Test.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot-XXX.*" bad_bot
SetEnvIfNoCase User-Agent "^ExaBotTest.*" bad_bot
SetEnvIfNoCase User-Agent "^ExactSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^exactseek.com.*" bad_bot
SetEnvIfNoCase User-Agent "^exooba/exooba crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Exploder.*" bad_bot
SetEnvIfNoCase User-Agent "^Express WebPictures.*" bad_bot
SetEnvIfNoCase User-Agent "^Extractor.*" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro.*" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE.*" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE.*" bad_bot
SetEnvIfNoCase User-Agent "^ez-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ezooms.*" bad_bot
SetEnvIfNoCase User-Agent "^Ezooms.*" bad_bot
SetEnvIfNoCase User-Agent "^f-bot test pilot.*" bad_bot
SetEnvIfNoCase User-Agent "^factbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Factbot.*" bad_bot
SetEnvIfNoCase User-Agent "^FairAd Client.*" bad_bot
SetEnvIfNoCase User-Agent "^falcon.*" bad_bot
SetEnvIfNoCase User-Agent "^Falconsbot.*" bad_bot
SetEnvIfNoCase User-Agent "^FAST Data Search Document Retriever.*" bad_bot
SetEnvIfNoCase User-Agent "^FAST ESP.*" bad_bot
SetEnvIfNoCase User-Agent "^fast-search-engine.*" bad_bot
SetEnvIfNoCase User-Agent "^fastbot crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^fastbot.de crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^FatBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Favcollector.*" bad_bot
SetEnvIfNoCase User-Agent "^Faviconizer.*" bad_bot
SetEnvIfNoCase User-Agent "^Favorites Sweeper.*" bad_bot
SetEnvIfNoCase User-Agent "^FDM 1.x.*" bad_bot
SetEnvIfNoCase User-Agent "^FDM 2.x.*" bad_bot
SetEnvIfNoCase User-Agent "^FDM 3.*" bad_bot
SetEnvIfNoCase User-Agent "^FDSE robot.*" bad_bot
SetEnvIfNoCase User-Agent "^FedContractorBot.*" bad_bot
SetEnvIfNoCase User-Agent "^fembot.*" bad_bot
SetEnvIfNoCase User-Agent "^Fetch API Request.*" bad_bot
SetEnvIfNoCase User-Agent "^fetch_ici.*" bad_bot
SetEnvIfNoCase User-Agent "^fgcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Filangy.*" bad_bot
SetEnvIfNoCase User-Agent "^FileHound.*" bad_bot
SetEnvIfNoCase User-Agent "^FindAnISP.com_ISP_Finder.*" bad_bot
SetEnvIfNoCase User-Agent "^findlinks.*" bad_bot
SetEnvIfNoCase User-Agent "^FindWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^Firebat.*" bad_bot
SetEnvIfNoCase User-Agent "^FirstGov.gov Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Flaming AttackBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Flamingo_SearchEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashCapture.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashGet.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashGet.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashGet WebWasher 3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^FlickySearchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Fluffy the spider.*" bad_bot
SetEnvIfNoCase User-Agent "^flunky.*" bad_bot
SetEnvIfNoCase User-Agent "^focused_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^FollowSite.*" bad_bot
SetEnvIfNoCase User-Agent "^Foobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Fooooo_Web_Video_Crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^Fopper.*" bad_bot
SetEnvIfNoCase User-Agent "^Forest Conservation Portal,.*" bad_bot
SetEnvIfNoCase User-Agent "^Forest Conservation Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Forests.org Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^FormulaFinderBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Forschungsportal.*" bad_bot
SetEnvIfNoCase User-Agent "^fr_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Francis.*" bad_bot
SetEnvIfNoCase User-Agent "^FreeWebMonitoring SiteChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^FreshDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^freshlinks.exe.*" bad_bot
SetEnvIfNoCase User-Agent "^FriendFeedBot.*" bad_bot
SetEnvIfNoCase User-Agent "^frodo.at.*" bad_bot
SetEnvIfNoCase User-Agent "^froGgle.*" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage.*" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage [NC,OR].*" bad_bot
SetEnvIfNoCase User-Agent "^Froola Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^FU-NBI/FU-NBI-.*" bad_bot
SetEnvIfNoCase User-Agent "^full_breadth_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^FunnelBack.*" bad_bot
SetEnvIfNoCase User-Agent "^FurlBot.*" bad_bot
SetEnvIfNoCase User-Agent "^G10-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Gaisbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Gaisbot.*" bad_bot
SetEnvIfNoCase User-Agent "^GaisBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GalaxyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^gazz.*" bad_bot
SetEnvIfNoCase User-Agent "^generate_infomine_category_classifiers.*" bad_bot
SetEnvIfNoCase User-Agent "^genevabot.*" bad_bot
SetEnvIfNoCase User-Agent "^genieBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GenieBotRD_SmallCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^Genieo.*" bad_bot
SetEnvIfNoCase User-Agent "^Geomaxenginebot.*" bad_bot
SetEnvIfNoCase User-Agent "^geometabot.*" bad_bot
SetEnvIfNoCase User-Agent "^GeonaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GeoVisu.*" bad_bot
SetEnvIfNoCase User-Agent "^GetHTMLContents.*" bad_bot
SetEnvIfNoCase User-Agent "^Getleft.*" bad_bot
SetEnvIfNoCase User-Agent "^GetLeft.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/2.11.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.3.3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.3.4.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.0.0.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.1.2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.2b (Portuguxeas).*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.2c.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5a.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b6.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b7.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5c.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5d.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5e.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/5.0beta1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/5.0beta2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetSmart.*" bad_bot
SetEnvIfNoCase User-Agent "^GetURL.rexx v1.05.*" bad_bot
SetEnvIfNoCase User-Agent "^GetWeb!.*" bad_bot
SetEnvIfNoCase User-Agent "^Giant.*" bad_bot
SetEnvIfNoCase User-Agent "^Giant.*" bad_bot
SetEnvIfNoCase User-Agent "^Giant.*" bad_bot
SetEnvIfNoCase User-Agent "^GigablastOpenSource.*" bad_bot
SetEnvIfNoCase User-Agent "^Gigabot.*" bad_bot
SetEnvIfNoCase User-Agent "^Gigabot.*" bad_bot
SetEnvIfNoCase User-Agent "^GigaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GigabotSiteSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^Girafabot.*" bad_bot
SetEnvIfNoCase User-Agent "^GleameBot.*" bad_bot
SetEnvIfNoCase User-Agent "^gnome-vfs.*" bad_bot
SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It.*" bad_bot
SetEnvIfNoCase User-Agent "^Go!Zilla.*" bad_bot
SetEnvIfNoCase User-Agent "^GoForIt.com.*" bad_bot
SetEnvIfNoCase User-Agent "^GOFORITBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^gold crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Goldfire Server.*" bad_bot
SetEnvIfNoCase User-Agent "^Golem/1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^GoodJelly.*" bad_bot
SetEnvIfNoCase User-Agent "^Gordon-College-Google-Mini.*" bad_bot
SetEnvIfNoCase User-Agent "^goroam.*" bad_bot
SetEnvIfNoCase User-Agent "^GoSeebot.*" bad_bot
SetEnvIfNoCase User-Agent "^gotit.*" bad_bot
SetEnvIfNoCase User-Agent "^Govbot.*" bad_bot
SetEnvIfNoCase User-Agent "^GPU p2p crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Grabber.*" bad_bot
SetEnvIfNoCase User-Agent "^Grabber.*" bad_bot
SetEnvIfNoCase User-Agent "^GrabNet.*" bad_bot
SetEnvIfNoCase User-Agent "^Grafula.*" bad_bot
SetEnvIfNoCase User-Agent "^grapeFX.*" bad_bot
SetEnvIfNoCase User-Agent "^grapeshot.*" bad_bot
SetEnvIfNoCase User-Agent "^GrapeshotCrawler/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^grbot.*" bad_bot
SetEnvIfNoCase User-Agent "^GreenYogi [ZSEBOT].*" bad_bot
SetEnvIfNoCase User-Agent "^Gromit.*" bad_bot
SetEnvIfNoCase User-Agent "^grub.*" bad_bot
SetEnvIfNoCase User-Agent "^grub crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^grub-client.*" bad_bot
SetEnvIfNoCase User-Agent "^Grub.*" bad_bot
SetEnvIfNoCase User-Agent "^Grubclient-.*" bad_bot
SetEnvIfNoCase User-Agent "^GrubNG.*" bad_bot
SetEnvIfNoCase User-Agent "^GruBot.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa-accuracyEval.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa-eusleg.*" bad_bot
SetEnvIfNoCase User-Agent "^GSLFbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Gulliver.*" bad_bot
SetEnvIfNoCase User-Agent "^GulperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GurujiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC BUSINESS crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC SEARCH BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC WEB crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC Weblink crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC WORLD LINKS.*" bad_bot
SetEnvIfNoCase User-Agent "^gvcbot.com.*" bad_bot
SetEnvIfNoCase User-Agent "^HappyFunBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Harvest.*" bad_bot
SetEnvIfNoCase User-Agent "^Harvest.*" bad_bot
SetEnvIfNoCase User-Agent "^Harvest/1.5.*" bad_bot
SetEnvIfNoCase User-Agent "^HarvestMan.*" bad_bot
SetEnvIfNoCase User-Agent "^Hatena Antenna.*" bad_bot
SetEnvIfNoCase User-Agent "^Hatena Antenna.*" bad_bot
SetEnvIfNoCase User-Agent "^Hawler.*" bad_bot
SetEnvIfNoCase User-Agent "^hcat.*" bad_bot
SetEnvIfNoCase User-Agent "^hclsreport-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^HD nutch agent.*" bad_bot
SetEnvIfNoCase User-Agent "^Header_Test_Client.*" bad_bot
SetEnvIfNoCase User-Agent "^healia/healia.*" bad_bot
SetEnvIfNoCase User-Agent "^Helix.*" bad_bot
SetEnvIfNoCase User-Agent "^here will be link to crawler site.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix bot.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix.*" bad_bot
SetEnvIfNoCase User-Agent "^Heritrix.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix/3.1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix/3.1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^hijbul-heritrix-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^HiScan.*" bad_bot
SetEnvIfNoCase User-Agent "^HiSoftware AccMonitor Server.*" bad_bot
SetEnvIfNoCase User-Agent "^HiSoftware AccVerify.*" bad_bot
SetEnvIfNoCase User-Agent "^hitcrawler_.*" bad_bot
SetEnvIfNoCase User-Agent "^hivaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^hivaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^hloader.*" bad_bot
SetEnvIfNoCase User-Agent "^HLoader.*" bad_bot
SetEnvIfNoCase User-Agent "^HMSEbot.*" bad_bot
SetEnvIfNoCase User-Agent "^HMView.*" bad_bot
SetEnvIfNoCase User-Agent "^hoge.*" bad_bot
SetEnvIfNoCase User-Agent "^holmes.*" bad_bot
SetEnvIfNoCase User-Agent "^HomePageSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^Hooblybot-Image.*" bad_bot
SetEnvIfNoCase User-Agent "^HooWWWer.*" bad_bot
SetEnvIfNoCase User-Agent "^Hostcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^HSFT - Link Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^HSFT - LVU Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^HSlide.*" bad_bot
SetEnvIfNoCase User-Agent "^ht://check.*" bad_bot
SetEnvIfNoCase User-Agent "^htdig.*" bad_bot
SetEnvIfNoCase User-Agent "^Html Link Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^HTMLParser.*" bad_bot
SetEnvIfNoCase User-Agent "^httplib.*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack.*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack.*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack [NC,OR].*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack 3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Huaweisymantecspider.*" bad_bot
SetEnvIfNoCase User-Agent "^hul-wax.*" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks.*" bad_bot
SetEnvIfNoCase User-Agent "^HyperEstraier.*" bad_bot
SetEnvIfNoCase User-Agent "^Hyperix.*" bad_bot
SetEnvIfNoCase User-Agent "^ia_archiver.*" bad_bot
SetEnvIfNoCase User-Agent "^ia_archiver.*" bad_bot
SetEnvIfNoCase User-Agent "^IAArchiver-.*" bad_bot
SetEnvIfNoCase User-Agent "^ibuena.*" bad_bot
SetEnvIfNoCase User-Agent "^iCab.*" bad_bot
SetEnvIfNoCase User-Agent "^ICDS-Ingestion.*" bad_bot
SetEnvIfNoCase User-Agent "^ichiro.*" bad_bot
SetEnvIfNoCase User-Agent "^iCopyright Conductor.*" bad_bot
SetEnvIfNoCase User-Agent "^IEAutoDiscovery.*" bad_bot
SetEnvIfNoCase User-Agent "^IECheck.*" bad_bot
SetEnvIfNoCase User-Agent "^iHWebChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^IIITBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^iim_405.*" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Iltrovatore-Setaccio.*" bad_bot
SetEnvIfNoCase User-Agent "^IlTrovatore-Setaccio.*" bad_bot
SetEnvIfNoCase User-Agent "^IlTrovatore.*" bad_bot
SetEnvIfNoCase User-Agent "^Image Stripper.*" bad_bot
SetEnvIfNoCase User-Agent "^Image Sucker.*" bad_bot
SetEnvIfNoCase User-Agent "^ImageBot.*" bad_bot
SetEnvIfNoCase User-Agent "^imagefortress.*" bad_bot
SetEnvIfNoCase User-Agent "^ImagesHereImagesThereImagesEverywhere.*" bad_bot
SetEnvIfNoCase User-Agent "^ImageVisu.*" bad_bot
SetEnvIfNoCase User-Agent "^imds_monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^imo-google-robot-intelink.*" bad_bot
SetEnvIfNoCase User-Agent "^inagist.com url crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^inagist.com url crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Industry Cortex Webcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Indy Library.*" bad_bot
SetEnvIfNoCase User-Agent "^Indy Library [NC,OR].*" bad_bot
SetEnvIfNoCase User-Agent "^indylabs_marius.*" bad_bot
SetEnvIfNoCase User-Agent "^InelaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Inet32 Ctrl.*" bad_bot
SetEnvIfNoCase User-Agent "^inetbot.*" bad_bot
SetEnvIfNoCase User-Agent "^info seeker.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoLink.*" bad_bot
SetEnvIfNoCase User-Agent "^infomine.ucr.edu.*" bad_bot
SetEnvIfNoCase User-Agent "^INFOMINE.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Informant.*" bad_bot
SetEnvIfNoCase User-Agent "^Infoseek Sidewinder.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoSeek Sidewinder.*" bad_bot
SetEnvIfNoCase User-Agent "^Infoseek SideWinder.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoTekies.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoUSABot.*" bad_bot
SetEnvIfNoCase User-Agent "^INGRID.*" bad_bot
SetEnvIfNoCase User-Agent "^INGRID/0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Inktomi.*" bad_bot
SetEnvIfNoCase User-Agent "^InsightsCollector.*" bad_bot
SetEnvIfNoCase User-Agent "^InsightsWorksBot.*" bad_bot
SetEnvIfNoCase User-Agent "^InspireBot.*" bad_bot
SetEnvIfNoCase User-Agent "^InsumaScout.*" bad_bot
SetEnvIfNoCase User-Agent "^Intelix.*" bad_bot
SetEnvIfNoCase User-Agent "^Intelliseek.*" bad_bot
SetEnvIfNoCase User-Agent "^InterGET.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja 4.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja 5.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja 6.0.*" bad_bot
SetEnvIfNoCase User-Agent "^INTERNET RADIO crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^InternetLinkAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Interseek.*" bad_bot
SetEnvIfNoCase User-Agent "^IOI.*" bad_bot
SetEnvIfNoCase User-Agent "^ip-web-crawler.com.*" bad_bot
SetEnvIfNoCase User-Agent "^IPAdd Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^IpselonBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ipselonbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Iria.*" bad_bot
SetEnvIfNoCase User-Agent "^IRLbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Iron33/1.0.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Isara/Isara-.*" bad_bot
SetEnvIfNoCase User-Agent "^iSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^iSiloX.*" bad_bot
SetEnvIfNoCase User-Agent "^IstellaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^its-learning crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^IU_CSCI_B659_class_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^iVia Page Fetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^iVia Site Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^iVia.*" bad_bot
SetEnvIfNoCase User-Agent "^JadynAve.*" bad_bot
SetEnvIfNoCase User-Agent "^JadynAveBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta.*" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta Commons-HttpClient.*" bad_bot
SetEnvIfNoCase User-Agent "^Jbot.*" bad_bot
SetEnvIfNoCase User-Agent "^JemmaTheTourist.*" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Jetbot.*" bad_bot
SetEnvIfNoCase User-Agent "^JetBrains Omea Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^JetCar.*" bad_bot
SetEnvIfNoCase User-Agent "^Jim.*" bad_bot
SetEnvIfNoCase User-Agent "^JoBo.*" bad_bot
SetEnvIfNoCase User-Agent "^JobSpider_BA.*" bad_bot
SetEnvIfNoCase User-Agent "^JOC.*" bad_bot
SetEnvIfNoCase User-Agent "^JOC Web Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^JoeDog.*" bad_bot
SetEnvIfNoCase User-Agent "^JoyScapeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^JSpyda.*" bad_bot
SetEnvIfNoCase User-Agent "^Junut Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^JustView.*" bad_bot
SetEnvIfNoCase User-Agent "^Jyxobot.*" bad_bot
SetEnvIfNoCase User-Agent "^K.S.Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^KakcleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^kalooga.*" bad_bot
SetEnvIfNoCase User-Agent "^KaloogaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^KATATUDO-Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^kbeta1.*" bad_bot
SetEnvIfNoCase User-Agent "^KeepNI web site monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin.Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Keybot Translation-Search-Machine.*" bad_bot
SetEnvIfNoCase User-Agent "^KeywenBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Keyword Density/0.9.*" bad_bot
SetEnvIfNoCase User-Agent "^Keyword.Density.*" bad_bot
SetEnvIfNoCase User-Agent "^kinjabot.*" bad_bot
SetEnvIfNoCase User-Agent "^Kitenga-crawler-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^KiwiStatus.*" bad_bot
SetEnvIfNoCase User-Agent "^kmbot-.*" bad_bot
SetEnvIfNoCase User-Agent "^kmccrew Bot Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Knight.*" bad_bot
SetEnvIfNoCase User-Agent "^KnowItAll.*" bad_bot
SetEnvIfNoCase User-Agent "^Knowledge Engine.*" bad_bot
SetEnvIfNoCase User-Agent "^Knowledge.com.*" bad_bot
SetEnvIfNoCase User-Agent "^KoepaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Koninklijke.*" bad_bot
SetEnvIfNoCase User-Agent "^KrOWLer.*" bad_bot
SetEnvIfNoCase User-Agent "^KSbot.*" bad_bot
SetEnvIfNoCase User-Agent "^kuloko-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^kulturarw3.*" bad_bot
SetEnvIfNoCase User-Agent "^KummHttp.*" bad_bot
SetEnvIfNoCase User-Agent "^Kurzor.*" bad_bot
SetEnvIfNoCase User-Agent "^Kyluka crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^L.webis.*" bad_bot
SetEnvIfNoCase User-Agent "^Labhoo.*" bad_bot
SetEnvIfNoCase User-Agent "^labourunions411.*" bad_bot
SetEnvIfNoCase User-Agent "^Lachesis.*" bad_bot
SetEnvIfNoCase User-Agent "^lachesis.*" bad_bot
SetEnvIfNoCase User-Agent "^Lament.*" bad_bot
SetEnvIfNoCase User-Agent "^LamerExterminator.*" bad_bot
SetEnvIfNoCase User-Agent "^LapozzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^larbin.*" bad_bot
SetEnvIfNoCase User-Agent "^larbin.*" bad_bot
SetEnvIfNoCase User-Agent "^LARBIN-EXPERIMENTAL.*" bad_bot
SetEnvIfNoCase User-Agent "^LBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LeapTag.*" bad_bot
SetEnvIfNoCase User-Agent "^LeechFTP.*" bad_bot
SetEnvIfNoCase User-Agent "^LeechGet.*" bad_bot
SetEnvIfNoCase User-Agent "^LetsCrawl.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LexxeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^lftp.*" bad_bot
SetEnvIfNoCase User-Agent "^libcrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^libiViaCore.*" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP.*" bad_bot
SetEnvIfNoCase User-Agent "^libwww-perl.*" bad_bot
SetEnvIfNoCase User-Agent "^likse.*" bad_bot
SetEnvIfNoCase User-Agent "^Linguee Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Link Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^Link Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^link_checker.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkAlarm.*" bad_bot
SetEnvIfNoCase User-Agent "^linkbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Linkbot.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkCheck by Siteimprove.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkCheck Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^linkdex.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkLint.*" bad_bot
SetEnvIfNoCase User-Agent "^linklooker.*" bad_bot
SetEnvIfNoCase User-Agent "^Linkman.*" bad_bot
SetEnvIfNoCase User-Agent "^Links SQL.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix.*" bad_bot
SetEnvIfNoCase User-Agent "^LinksManager.com_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkSweeper.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^LiteFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^LitlrBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Little Grabber at Skanktale.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Livelapbot.*" bad_bot
SetEnvIfNoCase User-Agent "^LM Harvester.*" bad_bot
SetEnvIfNoCase User-Agent "^LMQueueBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LNSpiderguy.*" bad_bot
SetEnvIfNoCase User-Agent "^LoadTimeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LocalcomBot.*" bad_bot
SetEnvIfNoCase User-Agent "^locust.*" bad_bot
SetEnvIfNoCase User-Agent "^LolongBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LookBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Lsearch/sondeur.*" bad_bot
SetEnvIfNoCase User-Agent "^lssbot.*" bad_bot
SetEnvIfNoCase User-Agent "^LT Scotland Checklink.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-request.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial/1.34.*" bad_bot
SetEnvIfNoCase User-Agent "^LWP::Simple.*" bad_bot
SetEnvIfNoCase User-Agent "^LWP/natweb-bad-link-mailer.*" bad_bot
SetEnvIfNoCase User-Agent "^Lycos_Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Lydia Entity Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^LynnBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Lytranslate.*" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net.*" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net.*" bad_bot
SetEnvIfNoCase User-Agent "^Magnet.*" bad_bot
SetEnvIfNoCase User-Agent "^magpie-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Magus Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.Ru_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.RU_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.Ru_Bot/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.Ru.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.RU.*" bad_bot
SetEnvIfNoCase User-Agent "^MAINSEEK_BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Mammoth.*" bad_bot
SetEnvIfNoCase User-Agent "^Map robot.*" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch.*" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch.*" bad_bot
SetEnvIfNoCase User-Agent "^MaSagool.*" bad_bot
SetEnvIfNoCase User-Agent "^masidani_bot_.*" bad_bot
SetEnvIfNoCase User-Agent "^Mass Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Mass Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Mass Downloader/2.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Mata Hari.*" bad_bot
SetEnvIfNoCase User-Agent "^Mata.Hari.*" bad_bot
SetEnvIfNoCase User-Agent "^matentzn at cs dot man dot ac dot uk.*" bad_bot
SetEnvIfNoCase User-Agent "^maxamine.com--robot.*" bad_bot
SetEnvIfNoCase User-Agent "^maxamine.com-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^maxomobot.*" bad_bot
SetEnvIfNoCase User-Agent "^McBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mediapartners-Google.*" bad_bot
SetEnvIfNoCase User-Agent "^medrabbit.*" bad_bot
SetEnvIfNoCase User-Agent "^Megite.*" bad_bot
SetEnvIfNoCase User-Agent "^MemacBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Memo.*" bad_bot
SetEnvIfNoCase User-Agent "^MendeleyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mercator-.*" bad_bot
SetEnvIfNoCase User-Agent "^mercuryboard_user_agent_sql_injection.nasl.*" bad_bot
SetEnvIfNoCase User-Agent "^metacarta.*" bad_bot
SetEnvIfNoCase User-Agent "^Metaeuro Web Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Metager2.*" bad_bot
SetEnvIfNoCase User-Agent "^metager2-verification-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^MetaGloss.*" bad_bot
SetEnvIfNoCase User-Agent "^metal crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^metaquerier.cs.uiuc.edu.*" bad_bot
SetEnvIfNoCase User-Agent "^METASpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Metaspinner.*" bad_bot
SetEnvIfNoCase User-Agent "^MetaURI API/2.0  metauri.com.*" bad_bot
SetEnvIfNoCase User-Agent "^MetaURI API/2.0 +metauri.com.*" bad_bot
SetEnvIfNoCase User-Agent "^MFcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^MFHttpScan.*" bad_bot
SetEnvIfNoCase User-Agent "^MIDown tool.*" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc.*" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc/4.2.*" bad_bot
SetEnvIfNoCase User-Agent "^mini-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^minibot.*" bad_bot
SetEnvIfNoCase User-Agent "^miniRank.*" bad_bot
SetEnvIfNoCase User-Agent "^Mirror.*" bad_bot
SetEnvIfNoCase User-Agent "^Missigua Locator.*" bad_bot
SetEnvIfNoCase User-Agent "^Missigua Locator.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister Pix II 2.01.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister Pix II 2.02a.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX version.dll.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister.PiX.*" bad_bot
SetEnvIfNoCase User-Agent "^Miva.*" bad_bot
SetEnvIfNoCase User-Agent "^MJ12bot.*" bad_bot
SetEnvIfNoCase User-Agent "^MJ12bot.*" bad_bot
SetEnvIfNoCase User-Agent "^mnoGoSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^MnoGoSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^mod_accessibility.*" bad_bot
SetEnvIfNoCase User-Agent "^moduna.com.*" bad_bot
SetEnvIfNoCase User-Agent "^moget.*" bad_bot
SetEnvIfNoCase User-Agent "^moget.*" bad_bot
SetEnvIfNoCase User-Agent "^moget/2.1.*" bad_bot
SetEnvIfNoCase User-Agent "^MojeekBot.*" bad_bot
SetEnvIfNoCase User-Agent "^MonkeyCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^MOSES.*" bad_bot
SetEnvIfNoCase User-Agent "^mowserbot.*" bad_bot
SetEnvIfNoCase User-Agent "^MQbot.*" bad_bot
SetEnvIfNoCase User-Agent "^MSE360.*" bad_bot
SetEnvIfNoCase User-Agent "^MSIndianWebcrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^MSMOBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^msnbot-products.*" bad_bot
SetEnvIfNoCase User-Agent "^MSNPTC.*" bad_bot
SetEnvIfNoCase User-Agent "^MSRBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^MSRBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^MT-Soft.*" bad_bot
SetEnvIfNoCase User-Agent "^MultiText.*" bad_bot
SetEnvIfNoCase User-Agent "^My_Little_SearchEngine_Project.*" bad_bot
SetEnvIfNoCase User-Agent "^my-heritrix-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^my-heritrix-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^MyApp.*" bad_bot
SetEnvIfNoCase User-Agent "^MYCOMPANYBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^mycrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^MyEngines-US-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^MyFamilyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Myra.*" bad_bot
SetEnvIfNoCase User-Agent "^nabot.*" bad_bot
SetEnvIfNoCase User-Agent "^nabot_.*" bad_bot
SetEnvIfNoCase User-Agent "^Najdi.si.*" bad_bot
SetEnvIfNoCase User-Agent "^Nambu.*" bad_bot
SetEnvIfNoCase User-Agent "^NAMEPROTECT.*" bad_bot
SetEnvIfNoCase User-Agent "^NASA Search.*" bad_bot
SetEnvIfNoCase User-Agent "^NatchCVS.*" bad_bot
SetEnvIfNoCase User-Agent "^National Park Service Dan Buan.*" bad_bot
SetEnvIfNoCase User-Agent "^naver.*" bad_bot
SetEnvIfNoCase User-Agent "^naverbookmarkcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NaverBot_dloader.*" bad_bot
SetEnvIfNoCase User-Agent "^NaverBot-.*" bad_bot
SetEnvIfNoCase User-Agent "^NaverBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Navroad.*" bad_bot
SetEnvIfNoCase User-Agent "^NearSite.*" bad_bot
SetEnvIfNoCase User-Agent "^NEC-MeshExplorer.*" bad_bot
SetEnvIfNoCase User-Agent "^NeoScioCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NerdByNature.Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^NerdyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Nerima-crawl-.*" bad_bot
SetEnvIfNoCase User-Agent "^Nessus.*" bad_bot
SetEnvIfNoCase User-Agent "^NESSUS::SOAP.*" bad_bot
SetEnvIfNoCase User-Agent "^nestReader.*" bad_bot
SetEnvIfNoCase User-Agent "^Net Vampire.*" bad_bot
SetEnvIfNoCase User-Agent "^Net Vampire.*" bad_bot
SetEnvIfNoCase User-Agent "^Net Vampire.*" bad_bot
SetEnvIfNoCase User-Agent "^Net Vampire/3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Net::Trackback.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.10.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.23.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.24.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.25.*" bad_bot
SetEnvIfNoCase User-Agent "^NetCarta CyberPilot Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^Netcraft.*" bad_bot
SetEnvIfNoCase User-Agent "^NetID.com Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic.*" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic.*" bad_bot
SetEnvIfNoCase User-Agent "^Netprospector.*" bad_bot
SetEnvIfNoCase User-Agent "^NetResearchServer.*" bad_bot
SetEnvIfNoCase User-Agent "^NetResearchServer.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSeer.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSeer crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSeer crawler/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^NetShift=.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSongBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Netsparker.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSrcherP.*" bad_bot
SetEnvIfNoCase User-Agent "^NetZIP.*" bad_bot
SetEnvIfNoCase User-Agent "^NewMedhunt.*" bad_bot
SetEnvIfNoCase User-Agent "^news bot .*" bad_bot
SetEnvIfNoCase User-Agent "^News_Search_App.*" bad_bot
SetEnvIfNoCase User-Agent "^NewsGatherer.*" bad_bot
SetEnvIfNoCase User-Agent "^Newsgroupreporter.*" bad_bot
SetEnvIfNoCase User-Agent "^NewsTroveBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NextGenSearchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^nextthing.org.*" bad_bot
SetEnvIfNoCase User-Agent "^NG.*" bad_bot
SetEnvIfNoCase User-Agent "^NG.*" bad_bot
SetEnvIfNoCase User-Agent "^nicebot.*" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO.*" bad_bot
SetEnvIfNoCase User-Agent "^niki-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^NimbleCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^nimbus-1.*" bad_bot
SetEnvIfNoCase User-Agent "^ninetowns.*" bad_bot
SetEnvIfNoCase User-Agent "^Ninja.*" bad_bot
SetEnvIfNoCase User-Agent "^NjuiceBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NLese.*" bad_bot
SetEnvIfNoCase User-Agent "^NLESE USEPA.*" bad_bot
SetEnvIfNoCase User-Agent "^Nogate.*" bad_bot
SetEnvIfNoCase User-Agent "^Norbert the Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^NoteworthyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NPbot.*" bad_bot
SetEnvIfNoCase User-Agent "^NPBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NPBot-.*" bad_bot
SetEnvIfNoCase User-Agent "^NRCan intranet crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NSDL_Search_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^nu_tch-princeton/Nu_tch.*" bad_bot
SetEnvIfNoCase User-Agent "^nuggetize.com BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^nuSearch Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^NuSearch Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Nusearch Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Nutch.*" bad_bot
SetEnvIfNoCase User-Agent "^nutch.*" bad_bot
SetEnvIfNoCase User-Agent "^nutch1/huntsman.*" bad_bot
SetEnvIfNoCase User-Agent "^NutchCVS.*" bad_bot
SetEnvIfNoCase User-Agent "^NutchOrg.*" bad_bot
SetEnvIfNoCase User-Agent "^NWSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Nymesis.*" bad_bot
SetEnvIfNoCase User-Agent "^nys-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^ObjectsSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^oBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Obvius external linkcheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Obvius external linkcheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Ocelli.*" bad_bot
SetEnvIfNoCase User-Agent "^Octopus.*" bad_bot
SetEnvIfNoCase User-Agent "^ODP entries t_st.*" bad_bot
SetEnvIfNoCase User-Agent "^oegp v..*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.4.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.6.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.7.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.9.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.3.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.4.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.5.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Navigator.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline.Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^OGspider.*" bad_bot
SetEnvIfNoCase User-Agent "^OmiExplorer_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^OmniExplorer_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^OmniFind.*" bad_bot
SetEnvIfNoCase User-Agent "^omnifind.*" bad_bot
SetEnvIfNoCase User-Agent "^OmniWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^OnetSzukaj.*" bad_bot
SetEnvIfNoCase User-Agent "^online link validator.*" bad_bot
SetEnvIfNoCase User-Agent "^OOZBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Openbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Openbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Openfind.*" bad_bot
SetEnvIfNoCase User-Agent "^Openfind data gatherer.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenIntelligenceData.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenISearch.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenLink Virtuoso RDF crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenSearchServer_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^OpiDig.*" bad_bot
SetEnvIfNoCase User-Agent "^optidiscover.*" bad_bot
SetEnvIfNoCase User-Agent "^optidiscover.*" bad_bot
SetEnvIfNoCase User-Agent "^Oracle Secure Enterprise Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Oracle Ultra Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Oracle Ultra Search.*" bad_bot
SetEnvIfNoCase User-Agent "^OrangeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ORISBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ornl_crawler_1.*" bad_bot
SetEnvIfNoCase User-Agent "^ORNL_Mercury.*" bad_bot
SetEnvIfNoCase User-Agent "^osis-project.jp.*" bad_bot
SetEnvIfNoCase User-Agent "^OsO.*" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot.*" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot.*" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxMelonBot.*" bad_bot
SetEnvIfNoCase User-Agent "^OWLER-BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^owsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ozelot.*" bad_bot
SetEnvIfNoCase User-Agent "^P3P Client.*" bad_bot
SetEnvIfNoCase User-Agent "^page_verifier.*" bad_bot
SetEnvIfNoCase User-Agent "^PageBitesHyperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Pagebull.*" bad_bot
SetEnvIfNoCase User-Agent "^PageDown.*" bad_bot
SetEnvIfNoCase User-Agent "^PageFetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^PageGrabber.*" bad_bot
SetEnvIfNoCase User-Agent "^PagePeeker.*" bad_bot
SetEnvIfNoCase User-Agent "^PageRank Monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^pamsnbot.htm.*" bad_bot
SetEnvIfNoCase User-Agent "^Panopy Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^panscient.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Pansophica.*" bad_bot
SetEnvIfNoCase User-Agent "^Papa Foto.*" bad_bot
SetEnvIfNoCase User-Agent "^PaperLiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^PaperLiBot/2.1.*" bad_bot
SetEnvIfNoCase User-Agent "^parasite.*" bad_bot
SetEnvIfNoCase User-Agent "^parsijoo.*" bad_bot
SetEnvIfNoCase User-Agent "^Pathtraq.*" bad_bot
SetEnvIfNoCase User-Agent "^Pattern.*" bad_bot
SetEnvIfNoCase User-Agent "^Patwebbot.*" bad_bot
SetEnvIfNoCase User-Agent "^pavuk.*" bad_bot
SetEnvIfNoCase User-Agent "^pavuk.*" bad_bot
SetEnvIfNoCase User-Agent "^PaxleFramework.*" bad_bot
SetEnvIfNoCase User-Agent "^PBBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^pcBrowser.*" bad_bot
SetEnvIfNoCase User-Agent "^pd-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^penthesila.*" bad_bot
SetEnvIfNoCase User-Agent "^perform_crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^PerMan.*" bad_bot
SetEnvIfNoCase User-Agent "^PerMan Surfer.*" bad_bot
SetEnvIfNoCase User-Agent "^personal ultimate crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^PHP version tracker.*" bad_bot
SetEnvIfNoCase User-Agent "^PHPCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^PhpDig.*" bad_bot
SetEnvIfNoCase User-Agent "^PicoSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^PIENO robot.*" bad_bot
SetEnvIfNoCase User-Agent "^pipBot.*" bad_bot
SetEnvIfNoCase User-Agent "^pipeLiner.*" bad_bot
SetEnvIfNoCase User-Agent "^Pita.*" bad_bot
SetEnvIfNoCase User-Agent "^pixfinder.*" bad_bot
SetEnvIfNoCase User-Agent "^PiyushBot.*" bad_bot
SetEnvIfNoCase User-Agent "^plaNETWORK Bot Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Plucker.*" bad_bot
SetEnvIfNoCase User-Agent "^Plukkie.*" bad_bot
SetEnvIfNoCase User-Agent "^Plumtree 6.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Pockey.*" bad_bot
SetEnvIfNoCase User-Agent "^Pockey-GetHTML.*" bad_bot
SetEnvIfNoCase User-Agent "^PoCoHTTP.*" bad_bot
SetEnvIfNoCase User-Agent "^pogodak.ba.*" bad_bot
SetEnvIfNoCase User-Agent "^Pogodak.co.yu.*" bad_bot
SetEnvIfNoCase User-Agent "^Poirot.*" bad_bot
SetEnvIfNoCase User-Agent "^polybot.*" bad_bot
SetEnvIfNoCase User-Agent "^Pompos.*" bad_bot
SetEnvIfNoCase User-Agent "^Poodle predictor.*" bad_bot
SetEnvIfNoCase User-Agent "^PopScreenBot.*" bad_bot
SetEnvIfNoCase User-Agent "^PostPost.*" bad_bot
SetEnvIfNoCase User-Agent "^PrivacyFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^ProjectWF-java-test-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14.*" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^Proxem WebSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^proximic.*" bad_bot
SetEnvIfNoCase User-Agent "^PROXY crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^psbot.*" bad_bot
SetEnvIfNoCase User-Agent "^PsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^psbot-page.*" bad_bot
SetEnvIfNoCase User-Agent "^psbot.*" bad_bot
SetEnvIfNoCase User-Agent "^PSS-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^psycheclone.*" bad_bot
SetEnvIfNoCase User-Agent "^pub-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^pucl.*" bad_bot
SetEnvIfNoCase User-Agent "^pulseBot (pulse Web Miner).*" bad_bot
SetEnvIfNoCase User-Agent "^Pump.*" bad_bot
SetEnvIfNoCase User-Agent "^PWeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Python-urllib.*" bad_bot
SetEnvIfNoCase User-Agent "^Python-urllib.*" bad_bot
SetEnvIfNoCase User-Agent "^pythonic-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^PythonWikipediaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^q1.*" bad_bot
SetEnvIfNoCase User-Agent "^QEAVis agent.*" bad_bot
SetEnvIfNoCase User-Agent "^QFKBot.*" bad_bot
SetEnvIfNoCase User-Agent "^qualidade.*" bad_bot
SetEnvIfNoCase User-Agent "^Qualidator.com Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^QuepasaCreep.*" bad_bot
SetEnvIfNoCase User-Agent "^QueryN Metasearch.*" bad_bot
SetEnvIfNoCase User-Agent "^QueryN.Metasearch.*" bad_bot
SetEnvIfNoCase User-Agent "^quest.durato.*" bad_bot
SetEnvIfNoCase User-Agent "^Quintura-Crw.*" bad_bot
SetEnvIfNoCase User-Agent "^QunarBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Qweery_robot.txt_CheckBot.*" bad_bot
SetEnvIfNoCase User-Agent "^QweeryBot.*" bad_bot
SetEnvIfNoCase User-Agent "^r2iBot.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_CommentReader.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_CommentReader.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_FeedFetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_VoteReader.*" bad_bot
SetEnvIfNoCase User-Agent "^RaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Radian6.*" bad_bot
SetEnvIfNoCase User-Agent "^radian6_linkcheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Radiation Retriever 1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^RAMPyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RankurBot/Rankur.*" bad_bot
SetEnvIfNoCase User-Agent "^Rational SiteCheck.*" bad_bot
SetEnvIfNoCase User-Agent "^RcStartBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload/4.0.0.40.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload/4.0.0.41.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload/4.0.0.42.*" bad_bot
SetEnvIfNoCase User-Agent "^Reaper.*" bad_bot
SetEnvIfNoCase User-Agent "^REBI-Shoveler.*" bad_bot
SetEnvIfNoCase User-Agent "^REBI-shoveler.*" bad_bot
SetEnvIfNoCase User-Agent "^Recorder.*" bad_bot
SetEnvIfNoCase User-Agent "^RedBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RedCarpet.*" bad_bot
SetEnvIfNoCase User-Agent "^ReGet.*" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey.*" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey Bait & Tackle/v1.01.*" bad_bot
SetEnvIfNoCase User-Agent "^research robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Riddler.*" bad_bot
SetEnvIfNoCase User-Agent "^RIIGHTBOT/RIIGHT-.*" bad_bot
SetEnvIfNoCase User-Agent "^RiseNetBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RiverGlassScanner.*" bad_bot
SetEnvIfNoCase User-Agent "^RiverglassScanner.*" bad_bot
SetEnvIfNoCase User-Agent "^RMA.*" bad_bot
SetEnvIfNoCase User-Agent "^RoboPal.*" bad_bot
SetEnvIfNoCase User-Agent "^Robosourcer.*" bad_bot
SetEnvIfNoCase User-Agent "^Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^ROBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^robot.*" bad_bot
SetEnvIfNoCase User-Agent "^robotek.*" bad_bot
SetEnvIfNoCase User-Agent "^robots.*" bad_bot
SetEnvIfNoCase User-Agent "^Robozilla.*" bad_bot
SetEnvIfNoCase User-Agent "^rogerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Rome Client.*" bad_bot
SetEnvIfNoCase User-Agent "^Rondello.*" bad_bot
SetEnvIfNoCase User-Agent "^Rotondo.*" bad_bot
SetEnvIfNoCase User-Agent "^Roverbot.*" bad_bot
SetEnvIfNoCase User-Agent "^RPT-HTTPClient.*" bad_bot
SetEnvIfNoCase User-Agent "^rtgibot.*" bad_bot
SetEnvIfNoCase User-Agent "^RufusBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Runnk online rss reader.*" bad_bot
SetEnvIfNoCase User-Agent "^Runnk RSS aggregator.*" bad_bot
SetEnvIfNoCase User-Agent "^s~stremor-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^S2Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^SafariBookmarkChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^SafetyNet Robot 0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^SaladSpoon/ShopSalad.*" bad_bot
SetEnvIfNoCase User-Agent "^Sapienti/Indexer.*" bad_bot
SetEnvIfNoCase User-Agent "^SBIder.*" bad_bot
SetEnvIfNoCase User-Agent "^SBL-BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^SCFCrawler/Nutch-1.8.*" bad_bot
SetEnvIfNoCase User-Agent "^Scich.*" bad_bot
SetEnvIfNoCase User-Agent "^ScientificCommons.org.*" bad_bot
SetEnvIfNoCase User-Agent "^ScollSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^ScooperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Scooter.*" bad_bot
SetEnvIfNoCase User-Agent "^ScoutJet.*" bad_bot
SetEnvIfNoCase User-Agent "^ScrapeBox.*" bad_bot
SetEnvIfNoCase User-Agent "^Scrapy.*" bad_bot
SetEnvIfNoCase User-Agent "^SCrawlTest.*" bad_bot
SetEnvIfNoCase User-Agent "^Scrubby.*" bad_bot
SetEnvIfNoCase User-Agent "^scSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Scumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Search Publisher.*" bad_bot
SetEnvIfNoCase User-Agent "^search x-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Search-Channel.*" bad_bot
SetEnvIfNoCase User-Agent "^Search-Engine-Studio.*" bad_bot
SetEnvIfNoCase User-Agent "^search.KumKie.com.*" bad_bot
SetEnvIfNoCase User-Agent "^search.msn.com/msnbot.htm.*" bad_bot
SetEnvIfNoCase User-Agent "^search.updated.com.*" bad_bot
SetEnvIfNoCase User-Agent "^search.usgs.gov.*" bad_bot
SetEnvIfNoCase User-Agent "^Searcharoo.NET.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchBlox.*" bad_bot
SetEnvIfNoCase User-Agent "^searchbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^searchengine.*" bad_bot
SetEnvIfNoCase User-Agent "^searchhippo.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchIt-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmarking.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmarks.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmee_v.*" bad_bot
SetEnvIfNoCase User-Agent "^Searchmee! Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmining.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchnowBot_v1.*" bad_bot
SetEnvIfNoCase User-Agent "^searchpreview.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchSpider.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchSpider.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SearQuBot/SearQuBot v1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^SEB Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Seekbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Seeker.lookseek.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SeeqBot.*" bad_bot
SetEnvIfNoCase User-Agent "^seeqpod-vertical-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Selflinkchecker.*" bad_bot
SetEnvIfNoCase User-Agent "^Semager.*" bad_bot
SetEnvIfNoCase User-Agent "^semanticdiscovery.*" bad_bot
SetEnvIfNoCase User-Agent "^Semantifire1.*" bad_bot
SetEnvIfNoCase User-Agent "^semisearch.*" bad_bot
SetEnvIfNoCase User-Agent "^SemrushBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SEOENGWorldBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SeznamBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SeznamBot/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^SeznamBot/3.2-test1.*" bad_bot
SetEnvIfNoCase User-Agent "^ShablastBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ShadowWebAnalyzer.*" bad_bot
SetEnvIfNoCase User-Agent "^Shareaza.*" bad_bot
SetEnvIfNoCase User-Agent "^Shelob.*" bad_bot
SetEnvIfNoCase User-Agent "^sherlock.*" bad_bot
SetEnvIfNoCase User-Agent "^ShopWiki.*" bad_bot
SetEnvIfNoCase User-Agent "^ShowLinks.*" bad_bot
SetEnvIfNoCase User-Agent "^ShowyouBot.*" bad_bot
SetEnvIfNoCase User-Agent "^siclab.*" bad_bot
SetEnvIfNoCase User-Agent "^silk.*" bad_bot
SetEnvIfNoCase User-Agent "^Siphon.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteArchive.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteCheck-sitecrawl by Siteimprove.com.*" bad_bot
SetEnvIfNoCase User-Agent "^sitecheck.internetseer.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteGuardBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteOrbiter.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSucker.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSweeper.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteXpert.*" bad_bot
SetEnvIfNoCase User-Agent "^SkimBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SkimWordsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SkreemRBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Skywalker.*" bad_bot
SetEnvIfNoCase User-Agent "^Sleipnir.*" bad_bot
SetEnvIfNoCase User-Agent "^slow-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch.*" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch.*" bad_bot
SetEnvIfNoCase User-Agent "^smart-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.76 (Win32; Apr 1 1999)" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.77 (Win32; Aug 17 1999)" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.77 (Win32; Feb 1 2000)" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.77 (Win32; Jun 19 2001)" bad_bot
SetEnvIfNoCase User-Agent "^Smarte Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^smartwit.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Snake.*" bad_bot
SetEnvIfNoCase User-Agent "^snap.com beta crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Snapbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SnapPreviewBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Snappy.*" bad_bot
SetEnvIfNoCase User-Agent "^snookit/Snookit.*" bad_bot
SetEnvIfNoCase User-Agent "^Snooper/b97_01.*" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy.*" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy.*" bad_bot
SetEnvIfNoCase User-Agent "^SocSciBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SOFT411 Directory.*" bad_bot
SetEnvIfNoCase User-Agent "^sogou.*" bad_bot
SetEnvIfNoCase User-Agent "^Sogou.*" bad_bot
SetEnvIfNoCase User-Agent "^sohu agent.*" bad_bot
SetEnvIfNoCase User-Agent "^sohu-search.*" bad_bot
SetEnvIfNoCase User-Agent "^Sokitomi crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^Solbot/1.0 LWP/5.07.*" bad_bot
SetEnvIfNoCase User-Agent "^sootle.*" bad_bot
SetEnvIfNoCase User-Agent "^Sosospider.*" bad_bot
SetEnvIfNoCase User-Agent "^Space Bison.*" bad_bot
SetEnvIfNoCase User-Agent "^Space Fung.*" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison.*" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison.*" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot.*" bad_bot
SetEnvIfNoCase User-Agent "^spanner.*" bad_bot
SetEnvIfNoCase User-Agent "^Spanner/1.0 (Linux 2.0.27 i586)" bad_bot
SetEnvIfNoCase User-Agent "^Spatineo Monitor Controller.*" bad_bot
SetEnvIfNoCase User-Agent "^Spatineo Serval Controller.*" bad_bot
SetEnvIfNoCase User-Agent "^Spatineo Serval GetMapBot.*" bad_bot
SetEnvIfNoCase User-Agent "^special_archiver.*" bad_bot
SetEnvIfNoCase User-Agent "^Speedy Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^SpeedySpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphere Scout.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphere Scout&v.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphider.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphider2.*" bad_bot
SetEnvIfNoCase User-Agent "^spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Spider.TerraNautic.net.*" bad_bot
SetEnvIfNoCase User-Agent "^SpiderEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^SpiderKU.*" bad_bot
SetEnvIfNoCase User-Agent "^SpiderMan.*" bad_bot
SetEnvIfNoCase User-Agent "^Spinn3r.*" bad_bot
SetEnvIfNoCase User-Agent "^Spinne.*" bad_bot
SetEnvIfNoCase User-Agent "^sportcrew-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^spyder3.microsys.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SQ Webscanner.*" bad_bot
SetEnvIfNoCase User-Agent "^sqlmap.*" bad_bot
SetEnvIfNoCase User-Agent "^Squid-Prefetch.*" bad_bot
SetEnvIfNoCase User-Agent "^SquidClamAV_Redirector.*" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm.*" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm.*" bad_bot
SetEnvIfNoCase User-Agent "^SrevBot.*" bad_bot
SetEnvIfNoCase User-Agent "^sslbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SSM Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^StackRambler.*" bad_bot
SetEnvIfNoCase User-Agent "^StarDownloader.*" bad_bot
SetEnvIfNoCase User-Agent "^statbot.*" bad_bot
SetEnvIfNoCase User-Agent "^statcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^statedept-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Steeler.*" bad_bot
SetEnvIfNoCase User-Agent "^STEGMANN-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^stero.*" bad_bot
SetEnvIfNoCase User-Agent "^Stripper.*" bad_bot
SetEnvIfNoCase User-Agent "^Stumbler.*" bad_bot
SetEnvIfNoCase User-Agent "^suchclip.*" bad_bot
SetEnvIfNoCase User-Agent "^Sucker.*" bad_bot
SetEnvIfNoCase User-Agent "^SumeetBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SumitBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SummizeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SummizeFeedReader.*" bad_bot
SetEnvIfNoCase User-Agent "^Sunrise XP.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^superbot.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot/3.0 (Win32)" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot/3.1 (Win32)" bad_bot
SetEnvIfNoCase User-Agent "^SuperHTTP.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperLumin Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperPagesBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Supybot.*" bad_bot
SetEnvIfNoCase User-Agent "^SURF.*" bad_bot
SetEnvIfNoCase User-Agent "^Surfbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SurfControl.*" bad_bot
SetEnvIfNoCase User-Agent "^SurveyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^suzuran.*" bad_bot
SetEnvIfNoCase User-Agent "^SWEBot.*" bad_bot
SetEnvIfNoCase User-Agent "^swish-e.*" bad_bot
SetEnvIfNoCase User-Agent "^SygolBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SynapticWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^Syntryx ANT Scout Chassis Pheromone.*" bad_bot
SetEnvIfNoCase User-Agent "^SystemSearch-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz.*" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4.*" bad_bot
SetEnvIfNoCase User-Agent "^T-H-U-N-D-E-R-S-T-O-N-E.*" bad_bot
SetEnvIfNoCase User-Agent "^Tailrank.*" bad_bot
SetEnvIfNoCase User-Agent "^tAkeOut.*" bad_bot
SetEnvIfNoCase User-Agent "^Talkro Web-Shot.*" bad_bot
SetEnvIfNoCase User-Agent "^TAMU_CRAWLER.*" bad_bot
SetEnvIfNoCase User-Agent "^TapuzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Tarantula.*" bad_bot
SetEnvIfNoCase User-Agent "^targetblaster.com.*" bad_bot
SetEnvIfNoCase User-Agent "^TargetYourNews.com bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TAUSDataBot.*" bad_bot
SetEnvIfNoCase User-Agent "^taxinomiabot.*" bad_bot
SetEnvIfNoCase User-Agent "^TeamSoft WinInet Component.*" bad_bot
SetEnvIfNoCase User-Agent "^Tecomi Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TeezirBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1590.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1634.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1718.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1820.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1847.*" bad_bot
SetEnvIfNoCase User-Agent "^TeleportPro.*" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft.*" bad_bot
SetEnvIfNoCase User-Agent "^Teradex Mapper.*" bad_bot
SetEnvIfNoCase User-Agent "^TERAGRAM_CRAWLER.*" bad_bot
SetEnvIfNoCase User-Agent "^TerrawizBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TestBot.*" bad_bot
SetEnvIfNoCase User-Agent "^testbot.*" bad_bot
SetEnvIfNoCase User-Agent "^testBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^testing of bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TextBot.*" bad_bot
SetEnvIfNoCase User-Agent "^thatrobotsite.com.*" bad_bot
SetEnvIfNoCase User-Agent "^The Dyslexalizer.*" bad_bot
SetEnvIfNoCase User-Agent "^The Intraformant.*" bad_bot
SetEnvIfNoCase User-Agent "^The.Intraformant.*" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad.*" bad_bot
SetEnvIfNoCase User-Agent "^Theophrastus.*" bad_bot
SetEnvIfNoCase User-Agent "^theusefulbot.*" bad_bot
SetEnvIfNoCase User-Agent "^TheUsefulbot_.*" bad_bot
SetEnvIfNoCase User-Agent "^ThumbBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Thumbnail.CZ robot.*" bad_bot
SetEnvIfNoCase User-Agent "^thumbshots-de-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^tigerbot.*" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TinEye.*" bad_bot
SetEnvIfNoCase User-Agent "^Titan.*" bad_bot
SetEnvIfNoCase User-Agent "^TITAN.*" bad_bot
SetEnvIfNoCase User-Agent "^to-dress_ru_bot_.*" bad_bot
SetEnvIfNoCase User-Agent "^to-night-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher.*" bad_bot
SetEnvIfNoCase User-Agent "^Topicalizer.*" bad_bot
SetEnvIfNoCase User-Agent "^topicblogs.*" bad_bot
SetEnvIfNoCase User-Agent "^Toplistbot.*" bad_bot
SetEnvIfNoCase User-Agent "^TopServer PHP.*" bad_bot
SetEnvIfNoCase User-Agent "^topyx-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Touche.*" bad_bot
SetEnvIfNoCase User-Agent "^TourlentaScanner.*" bad_bot
SetEnvIfNoCase User-Agent "^TPSystem.*" bad_bot
SetEnvIfNoCase User-Agent "^TRAAZI.*" bad_bot
SetEnvIfNoCase User-Agent "^TranSGeniKBot.*" bad_bot
SetEnvIfNoCase User-Agent "^travel-search.*" bad_bot
SetEnvIfNoCase User-Agent "^TravelBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TravelLazerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Treezy.*" bad_bot
SetEnvIfNoCase User-Agent "^TREX.*" bad_bot
SetEnvIfNoCase User-Agent "^TridentSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Trovator.*" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^tScholarsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TSM Translation-Search-Machine.*" bad_bot
SetEnvIfNoCase User-Agent "^TsWebBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TulipChain.*" bad_bot
SetEnvIfNoCase User-Agent "^turingos.*" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5.*" bad_bot
SetEnvIfNoCase User-Agent "^TutorGigBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TweetedTimes Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TweetmemeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TwengaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TwengaBot-Discover.*" bad_bot
SetEnvIfNoCase User-Agent "^TwengaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Twiceler.*" bad_bot
SetEnvIfNoCase User-Agent "^Twikle.*" bad_bot
SetEnvIfNoCase User-Agent "^twinuffbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Twisted PageGetter.*" bad_bot
SetEnvIfNoCase User-Agent "^Twitturls.*" bad_bot
SetEnvIfNoCase User-Agent "^Twitturly .*" bad_bot
SetEnvIfNoCase User-Agent "^TygoBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TygoProwler.*" bad_bot
SetEnvIfNoCase User-Agent "^Typhoeus.*" bad_bot
SetEnvIfNoCase User-Agent "^U.S. Government Printing Office.*" bad_bot
SetEnvIfNoCase User-Agent "^uberbot.*" bad_bot
SetEnvIfNoCase User-Agent "^ucb-nutch.*" bad_bot
SetEnvIfNoCase User-Agent "^UdmSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^UFAM-crawler-.*" bad_bot
SetEnvIfNoCase User-Agent "^Ultraseek.*" bad_bot
SetEnvIfNoCase User-Agent "^UnChaos.*" bad_bot
SetEnvIfNoCase User-Agent "^unchaos_crawler_.*" bad_bot
SetEnvIfNoCase User-Agent "^UnisterBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Unitek UniEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^UniversalSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^UnwindFetchor.*" bad_bot
SetEnvIfNoCase User-Agent "^UnwindFetchor.*" bad_bot
SetEnvIfNoCase User-Agent "^UofTDB_experiment.*" bad_bot
SetEnvIfNoCase User-Agent "^updated.*" bad_bot
SetEnvIfNoCase User-Agent "^URL Control.*" bad_bot
SetEnvIfNoCase User-Agent "^url_gather.*" bad_bot
SetEnvIfNoCase User-Agent "^URL-Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^URLAppendBot.*" bad_bot
SetEnvIfNoCase User-Agent "^URLBlaze.*" bad_bot
SetEnvIfNoCase User-Agent "^urlchecker.*" bad_bot
SetEnvIfNoCase User-Agent "^urlck/1.2.3.*" bad_bot
SetEnvIfNoCase User-Agent "^UrlDispatcher.*" bad_bot
SetEnvIfNoCase User-Agent "^URLSpiderPro.*" bad_bot
SetEnvIfNoCase User-Agent "^URLy Warning.*" bad_bot
SetEnvIfNoCase User-Agent "^URLy.Warning.*" bad_bot
SetEnvIfNoCase User-Agent "^USAF AFKN K2SPIDER.*" bad_bot
SetEnvIfNoCase User-Agent "^usasearch.*" bad_bot
SetEnvIfNoCase User-Agent "^USS-Cosmix.*" bad_bot
SetEnvIfNoCase User-Agent "^USyd-NLP-Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Vacobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vacuum.*" bad_bot
SetEnvIfNoCase User-Agent "^VadixBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vagabondo.*" bad_bot
SetEnvIfNoCase User-Agent "^Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^Valkyrie/1.0 libwww-perl/0.40" bad_bot
SetEnvIfNoCase User-Agent "^vBSEO.*" bad_bot
SetEnvIfNoCase User-Agent "^VCI.*" bad_bot
SetEnvIfNoCase User-Agent "^VCI WebViewer VCI WebViewer Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^VerbstarBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VeriCiteCrawler/Nutch-2.2.1" bad_bot
SetEnvIfNoCase User-Agent "^Verifactrola.*" bad_bot
SetEnvIfNoCase User-Agent "^Verity-URL-Gateway.*" bad_bot
SetEnvIfNoCase User-Agent "^Vermut.*" bad_bot
SetEnvIfNoCase User-Agent "^vermut.*" bad_bot
SetEnvIfNoCase User-Agent "^versus crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^versus.integis.ch.*" bad_bot
SetEnvIfNoCase User-Agent "^viasarchivinginformation.html.*" bad_bot
SetEnvIfNoCase User-Agent "^VIP.*" bad_bot
SetEnvIfNoCase User-Agent "^VIPr.*" bad_bot
SetEnvIfNoCase User-Agent "^virus_detector.*" bad_bot
SetEnvIfNoCase User-Agent "^virus-detector.*" bad_bot
SetEnvIfNoCase User-Agent "^VisBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vishal For CLIA.*" bad_bot
SetEnvIfNoCase User-Agent "^VisWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^Vital Search'n Urchin.*" bad_bot
SetEnvIfNoCase User-Agent "^vlad.*" bad_bot
SetEnvIfNoCase User-Agent "^vlsearch.*" bad_bot
SetEnvIfNoCase User-Agent "^VMBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VocusBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VoidEYE.*" bad_bot
SetEnvIfNoCase User-Agent "^VoilaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VoilaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vortex.*" bad_bot
SetEnvIfNoCase User-Agent "^voyager-hc.*" bad_bot
SetEnvIfNoCase User-Agent "^voyager-partner-deep.*" bad_bot
SetEnvIfNoCase User-Agent "^voyager.*" bad_bot
SetEnvIfNoCase User-Agent "^VSE.*" bad_bot
SetEnvIfNoCase User-Agent "^vspider.*" bad_bot
SetEnvIfNoCase User-Agent "^W3C_Unicorn.*" bad_bot
SetEnvIfNoCase User-Agent "^W3C-WebCon.*" bad_bot
SetEnvIfNoCase User-Agent "^w3m.*" bad_bot
SetEnvIfNoCase User-Agent "^w3search.*" bad_bot
SetEnvIfNoCase User-Agent "^wacbot.*" bad_bot
SetEnvIfNoCase User-Agent "^wastrix.*" bad_bot
SetEnvIfNoCase User-Agent "^wastrix.*" bad_bot
SetEnvIfNoCase User-Agent "^Water Conserve Portal.*" bad_bot
SetEnvIfNoCase User-Agent "^Water Conserve Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^WatzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^wauuu engine/Wauuu.*" bad_bot
SetEnvIfNoCase User-Agent "^Wavefire.*" bad_bot
SetEnvIfNoCase User-Agent "^Waypath.*" bad_bot
SetEnvIfNoCase User-Agent "^Wazzup.*" bad_bot
SetEnvIfNoCase User-Agent "^Wazzup1.0.4800.*" bad_bot
SetEnvIfNoCase User-Agent "^wbdbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Web CEO Online robot.*" bad_bot
SetEnvIfNoCase User-Agent "^web crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Image Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Link Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Magnet.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Site Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Sucker.*" bad_bot
SetEnvIfNoCase User-Agent "^web-agent.*" bad_bot
SetEnvIfNoCase User-Agent "^Web-Sniffer.*" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^WebaltBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto.*" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto.*" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto/3.40.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit.*" bad_bot
SetEnvIfNoCase User-Agent "^webbandit.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit/3.50.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^webbot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBot.*" bad_bot
SetEnvIfNoCase User-Agent "^webbul-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCapture.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCapture 2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^webcheck 1.*" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com.*" bad_bot
SetEnvIfNoCase User-Agent "^webcollage.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v.2.2.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.5.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.6.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.7a.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.8.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.2a.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopy.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCorp.*" bad_bot
SetEnvIfNoCase User-Agent "^webcrawl.net.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^webcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCrawler/3.0 Robot libwww/5.0a.*" bad_bot
SetEnvIfNoCase User-Agent "^WebDownloader for X.*" bad_bot
SetEnvIfNoCase User-Agent "^Webdup.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer.*" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^webfetch.*" bad_bot
SetEnvIfNoCase User-Agent "^webfetch.*" bad_bot
SetEnvIfNoCase User-Agent "^WebFetcher/0.8,.*" bad_bot
SetEnvIfNoCase User-Agent "^WebGather.*" bad_bot
SetEnvIfNoCase User-Agent "^WebGo IS.*" bad_bot
SetEnvIfNoCase User-Agent "^webGobbler.*" bad_bot
SetEnvIfNoCase User-Agent "^WebImages.*" bad_bot
SetEnvIfNoCase User-Agent "^Webinator-search2.fasthealth.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Webinator-WBI.*" bad_bot
SetEnvIfNoCase User-Agent "^WebIndex.*" bad_bot
SetEnvIfNoCase User-Agent "^WebIndexer.*" bad_bot
SetEnvIfNoCase User-Agent "^weblayers/0.0.*" bad_bot
SetEnvIfNoCase User-Agent "^WebLeacher.*" bad_bot
SetEnvIfNoCase User-Agent "^WeblexBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebLinker/0.0 libwww-perl/0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^webLyzard.*" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterCoffee.*" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorld Extractor.*" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebMiner.*" bad_bot
SetEnvIfNoCase User-Agent "^WebMiner.*" bad_bot
SetEnvIfNoCase User-Agent "^WebMoose/0.0.0000.*" bad_bot
SetEnvIfNoCase User-Agent "^WeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebPix.*" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebRipper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger 1.20b.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger 1.20j.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger 1.20k.*" bad_bot
SetEnvIfNoCase User-Agent "^Webscan.*" bad_bot
SetEnvIfNoCase User-Agent "^websearchbench.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSite.*" bad_bot
SetEnvIfNoCase User-Agent "^Website Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Website Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^Website eXtractor.*" bad_bot
SetEnvIfNoCase User-Agent "^Website Quester.*" bad_bot
SetEnvIfNoCase User-Agent "^Website.Quester.*" bad_bot
SetEnvIfNoCase User-Agent "^websitemirror.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSpear.*" bad_bot
SetEnvIfNoCase User-Agent "^websphinx.test.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster.Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.03.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.10.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.12.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.13.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.15.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.16.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.19.*" bad_bot
SetEnvIfNoCase User-Agent "^WebTrafficExpress.*" bad_bot
SetEnvIfNoCase User-Agent "^WebTrends Link Analyzer.*" bad_bot
SetEnvIfNoCase User-Agent "^WebVac.*" bad_bot
SetEnvIfNoCase User-Agent "^webwalk.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWalker/1.10.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWasher.*" bad_bot
SetEnvIfNoCase User-Agent "^Webwasher.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWatch.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWhacker.*" bad_bot
SetEnvIfNoCase User-Agent "^WebXM.*" bad_bot
SetEnvIfNoCase User-Agent "^WebZIP.*" bad_bot
SetEnvIfNoCase User-Agent "^Weddings.info Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^wenbin/search.*" bad_bot
SetEnvIfNoCase User-Agent "^WEP Search.*" bad_bot
SetEnvIfNoCase User-Agent "^WEPA.*" bad_bot
SetEnvIfNoCase User-Agent "^WeRelateBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget.*" bad_bot
SetEnvIfNoCase User-Agent "^wget.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.4.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.7.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.1+cvs.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.9-beta.*" bad_bot
SetEnvIfNoCase User-Agent "^Whacker.*" bad_bot
SetEnvIfNoCase User-Agent "^Whirlpool Web Engine.*" bad_bot
SetEnvIfNoCase User-Agent "^WhoWhere Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Widow.*" bad_bot
SetEnvIfNoCase User-Agent "^WikiaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Wikio.*" bad_bot
SetEnvIfNoCase User-Agent "^wikiwix-bot-.*" bad_bot
SetEnvIfNoCase User-Agent "^WinHTTP Example.*" bad_bot
SetEnvIfNoCase User-Agent "^WinHttp.WinHttpRequest..*" bad_bot
SetEnvIfNoCase User-Agent "^WIRE.*" bad_bot
SetEnvIfNoCase User-Agent "^wired-digital-newsbot/1.5.*" bad_bot
SetEnvIfNoCase User-Agent "^WISEbot.*" bad_bot
SetEnvIfNoCase User-Agent "^WISENutbot.*" bad_bot
SetEnvIfNoCase User-Agent "^wish-la.*" bad_bot
SetEnvIfNoCase User-Agent "^wish-project.*" bad_bot
SetEnvIfNoCase User-Agent "^wisponbot.*" bad_bot
SetEnvIfNoCase User-Agent "^WMCAI-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^wminer.*" bad_bot
SetEnvIfNoCase User-Agent "^WMSBot.*" bad_bot
SetEnvIfNoCase User-Agent "^woriobot.*" bad_bot
SetEnvIfNoCase User-Agent "^worldshop.*" bad_bot
SetEnvIfNoCase User-Agent "^WorQmada.*" bad_bot
SetEnvIfNoCase User-Agent "^Wotbox.*" bad_bot
SetEnvIfNoCase User-Agent "^wume_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^WWW Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E.*" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Mechanize.*" bad_bot
SetEnvIfNoCase User-Agent "^WWWOFFLE.*" bad_bot
SetEnvIfNoCase User-Agent "^wwwrobot.*" bad_bot
SetEnvIfNoCase User-Agent "^wwwster.*" bad_bot
SetEnvIfNoCase User-Agent "^WWWWanderer v3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^wwwxref.*" bad_bot
SetEnvIfNoCase User-Agent "^Wysigot.*" bad_bot
SetEnvIfNoCase User-Agent "^X-clawler.*" bad_bot
SetEnvIfNoCase User-Agent "^X-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon WebSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon WebSpider 2.5.b3.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon_WebSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu's.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu's Link Sleuth 1.1c.*" bad_bot
SetEnvIfNoCase User-Agent "^Xerka MetaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Xerka WebBot.*" bad_bot
SetEnvIfNoCase User-Agent "^XGET/0.7.*" bad_bot
SetEnvIfNoCase User-Agent "^xirq.*" bad_bot
SetEnvIfNoCase User-Agent "^XmarksFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^xqrobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BRI.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BRJ/YATS crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BRW.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BSC.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-PSC.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-SRD.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!TunnelPro.*" bad_bot
SetEnvIfNoCase User-Agent "^yacy.net.*" bad_bot
SetEnvIfNoCase User-Agent "^yacybot.*" bad_bot
SetEnvIfNoCase User-Agent "^Yandex.*" bad_bot
SetEnvIfNoCase User-Agent "^YandexBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YandexSomething.*" bad_bot
SetEnvIfNoCase User-Agent "^Yanga WorldSearch Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Yanga WorldSearch Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^yarienavoir.net.*" bad_bot
SetEnvIfNoCase User-Agent "^Yasaklibot.*" bad_bot
SetEnvIfNoCase User-Agent "^yBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YebolBot.*" bad_bot
SetEnvIfNoCase User-Agent "^yellowJacket.*" bad_bot
SetEnvIfNoCase User-Agent "^yes.*" bad_bot
SetEnvIfNoCase User-Agent "^YesupBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YesupBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Yeti.*" bad_bot
SetEnvIfNoCase User-Agent "^Yeti.*" bad_bot
SetEnvIfNoCase User-Agent "^YioopBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YioopBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YioopBot.*" bad_bot
SetEnvIfNoCase User-Agent "^yolinkBot.*" bad_bot
SetEnvIfNoCase User-Agent "^yoogliFetchAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Yoono.*" bad_bot
SetEnvIfNoCase User-Agent "^yoono.*" bad_bot
SetEnvIfNoCase User-Agent "^Yoriwa.*" bad_bot
SetEnvIfNoCase User-Agent "^YottaCars_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^you-dir.*" bad_bot
SetEnvIfNoCase User-Agent "^Z-Add Link Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^zagrebin.*" bad_bot
SetEnvIfNoCase User-Agent "^Zao.*" bad_bot
SetEnvIfNoCase User-Agent "^zedzo.digest.*" bad_bot
SetEnvIfNoCase User-Agent "^zedzo.validate.*" bad_bot
SetEnvIfNoCase User-Agent "^zermelo.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 11389 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 11652 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 18018 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 26378 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 30747 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 32297 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 39206 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 41641 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 44238 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 51070 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 51674 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 51837 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 63567 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 6694 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 82016 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 82900 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 84842 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 90872 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 94934 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 95245 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 95351 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 97371 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus Link Scout.*" bad_bot
SetEnvIfNoCase User-Agent "^zibber-v.*" bad_bot
SetEnvIfNoCase User-Agent "^zimeno.*" bad_bot
SetEnvIfNoCase User-Agent "^Zing-BottaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ZipppBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ZoomSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Zotag Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Zotag Search.*" bad_bot
SetEnvIfNoCase User-Agent "^ZuiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ZumBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Zyborg.*" bad_bot
SetEnvIfNoCase User-Agent "^ZyBorg.*" bad_bot
SetEnvIfNoCase User-Agent "^Zyte.*" bad_bot
<Limit GET POST PUT>
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot
</Limit>
</IfModule>
# Fine blocco Bad Bots

Ovviamente il codice è implementabile aggiungendo altri USER AGENT semplicemente aggiungendo gli adeguati riferimenti.

Utilizzare 6G Firewall (nel file .htaccess)

In alternativa alla pedissequa elencazione degli User-Agent possiamo utilizzare qualcosa di più evoluto.

Nello specifico, al posto del codice presentato in precedenza, possiamo arricchire il nostro file .htaccess attraverso il codice sviluppato all’interno del progetto 6G Firewall. Si tratta di alcune direttive plug-n-play in grado di proteggere il nostro sito da una serie di minacce tra cui, appunto, i Bad Bots. Per vostra comodità riporto il codice completo qui sotto:

# 6G FIREWALL/BLACKLIST

# 6G:[QUERY STRINGS]
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{QUERY_STRING} (eval() [NC,OR]
  RewriteCond %{QUERY_STRING} (127.0.0.1) [NC,OR]
  RewriteCond %{QUERY_STRING} ([a-z0-9]{2000,}) [NC,OR]
  RewriteCond %{QUERY_STRING} (javascript:)(.*)(;) [NC,OR]
  RewriteCond %{QUERY_STRING} (base64_encode)(.*)(() [NC,OR]
  RewriteCond %{QUERY_STRING} (GLOBALS|REQUEST)(=|[|%) [NC,OR]
  RewriteCond %{QUERY_STRING} (<|%3C)(.*)script(.*)(>|%3) [NC,OR]
  RewriteCond %{QUERY_STRING} (\|...|../|~|`|<|>||) [NC,OR]
  RewriteCond %{QUERY_STRING} (boot.ini|etc/passwd|self/environ) [NC,OR]
  RewriteCond %{QUERY_STRING} (thumbs?(_editor|open)?|tim(thumb)?).php [NC,OR]
  RewriteCond %{QUERY_STRING} ('|")(.*)(drop|insert|md5|select|union) [NC]
  RewriteRule .* - [F]
</IfModule>

# 6G:[REQUEST METHOD]
<IfModule mod_rewrite.c>
  RewriteCond %{REQUEST_METHOD} ^(connect|debug|move|put|trace|track) [NC]
  RewriteRule .* - [F]
</IfModule>

# 6G:[REFERRERS]
<IfModule mod_rewrite.c>
  RewriteCond %{HTTP_REFERER} ([a-z0-9]{2000,}) [NC,OR]
  RewriteCond %{HTTP_REFERER} (semalt.com|todaperfeita) [NC]
  RewriteRule .* - [F]
</IfModule>

# 6G:[REQUEST STRINGS]
<IfModule mod_alias.c>
  RedirectMatch 403 (?i)([a-z0-9]{2000,})
  RedirectMatch 403 (?i)(https?|ftp|php):/
  RedirectMatch 403 (?i)(base64_encode)(.*)(()
  RedirectMatch 403 (?i)(=\'|=\%27|/\'/?).
  RedirectMatch 403 (?i)/($(&)?|*|"|.|,|&|&?)/?$
  RedirectMatch 403 (?i)({0}|(/(|...|+++|\"\")
  RedirectMatch 403 (?i)(~|`|<|>|:|;|,|%|\|{|}|[|]||)
  RedirectMatch 403 (?i)/(=|$&|_mm|cgi-|muieblack)
  RedirectMatch 403 (?i)(&pws=0|_vti_|(null)|{$itemURL}|echo(.*)kae|etc/passwd|eval(|self/environ)
  RedirectMatch 403 (?i).(aspx?|bash|bak?|cfg|cgi|dll|exe|git|hg|ini|jsp|log|mdb|out|sql|svn|swp|tar|rar|rdf)$
  RedirectMatch 403 (?i)/(^$|(wp-)?config|mobiquo|phpinfo|shell|sqlpatch|thumb|thumb_editor|thumbopen|timthumb|webshell).php
</IfModule>

# 6G:[USER AGENTS]
<IfModule mod_setenvif.c>
  SetEnvIfNoCase User-Agent ([a-z0-9]{2000,}) bad_bot
  SetEnvIfNoCase User-Agent (archive.org|binlar|casper|checkpriv|choppy|clshttp|cmsworld|diavol|dotbot|extract|feedfinder|flicky|g00g1e|harvest|heritrix|httrack|kmccrew|loader|miner|nikto|nutch|planetwork|postrank|purebot|pycurl|python|seekerspider|siclab|skygrid|sqlmap|sucker|turnit|vikspider|winhttp|xxxyy|youda|zmeu|zune) bad_bot
  
  # Apache < 2.3
  <IfModule !mod_authz_core.c>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
  </IfModule>

  # Apache >= 2.3
  <IfModule mod_authz_core.c>
    <RequireAll>
      Require all Granted
      Require not env bad_bot
    </RequireAll>
  </IfModule>
</IfModule>

# 6G:[BAD IPS]
<Limit GET HEAD OPTIONS POST PUT>
  Order Allow,Deny
  Allow from All
  # uncomment/edit/repeat next line to block IPs
  # Deny from 123.456.789
</Limit>

Implementare una trappola per identificare i Bad Bots

Per finire, qualora gli stratagemmi posti in atto sino ad ora non siano stati del tutto risolutivi, si può pensare di implementare una sorta di honeypot (letteralmente "trappola di miele") mediante script ad hoc come Black Hole (clicca qui per la versione PHP o qui per il plugin WordPress) o qualcosa di similare.

Ma come funzionano queste trappole? Semplice: questi sistemi di protezione generano, all’interno del sito, dei link "speciali" inaccessibili sia agli utenti umani (perché presenti nel sorgente della pagina ma nascosti agli occhi del visitatore) che ai "bot buoni" (perché a questi ultimi proibiti mediante apposite regole nel file robots.txt), con la conseguenza che ogni visita a queste risorse potrà essere frutto solo dell’operato di un qualche bad bot. Una volta identificato l’ip, questo verà bannato dal sito impedendone, quindi, future scansioni. Semplice ed efficace.