标签：蜘蛛

为什么屏蔽垃圾蜘蛛？ 屏蔽垃圾蜘蛛主要有以下几个原因： 节省服务器资源：垃圾蜘蛛通常会频繁抓取网站内容，消耗大量的服务器资源，导致正常用户访问速度变慢。 保护网站安全：一些垃圾蜘蛛可能会尝试扫描网站漏洞，增加网站被攻击的风险。 提高SEO效果：垃圾蜘蛛抓取的内容通常不会对网站的SEO有正面影响，反而可能会影响搜索引擎对网站的评价。 减少日志文件大小：垃圾蜘蛛的抓取行为会在服务器日志中留下大量无用的记录，占用存储空间。 通过屏蔽这些垃圾蜘蛛，你可以有效地保护你的网站资源，提升用户体验，并减少潜在的安全风险。 宝塔中提供了付费版和免费版的Nginx防火墙。我们可以利用 User-Agent 来屏蔽掉许多无关紧要的蜘蛛爬虫，并且防止 SQL 注入和菜刀一句话的入侵。 然而，免费版的 nginx 防火墙规则相对较少，种类也不够多样化。因此，我们将分享一些来自付费版的规则，您可以手动逐条添加到免费版的防火墙中。遗憾的是，免费版没有导入功能。 安装Nginx免费防火墙 免费防火墙防御效果 添加过滤垃圾蜘蛛爬虫规则 添加垃圾蜘蛛规则，包括常见的垃圾蜘蛛和AI爬虫 (www\.seokicks\.de|YYSpider|Mattermost|Discord|CCBot|RepoLookoutBot|FBexternalhit|serpstatbot|Pinterestbot|SurdotlyBot|DataForSeoBot|DigExt|HttpClient|MJ12bot|heritrix|Ezooms|FlightDeckReports|LingueeBot|Web-Crawler|WellKnownBot|Yellowbrandprotectionbot|ev-crawler|NE Crawler|Facebot|facebookexternalhit|meta-externalagent|facebookscraper|facebookexternalhit|GrapeshotCrawler|SemrushBot|DotBot|MegaIndex\.ru|MauiBot|AhrefsBot|BLEXBot|HubSpotCrawler|CriteoBot|YaK|Mail\.RU_Bot|Barkrowler|vxiaotou-spider|telegram|dingtalk|DuckDuckGo|applebot|webprosbot|AwarioBot|Amazonbot|AmazonAdBot|YouBot|P/2\.0|YandexBot|Slurp|AdsBot-Google|Googlebot-Image|Googlebot-News|Googlebot-Video|msnbot|Twitterbot|Slackbot|LinkedInBot|NaverBot|Yeti|Daumoa|Teoma|ScoutJet|Exabot|Cliqzbot|SeznamBot|PaperLiBot|SputnikBot|Qwantify|Steeler|Gigabot|Scooter|TurnitinBot|Scrapy|Diffbot|Kozmos|KaloogaBot|LSSRocketCrawler|Nutch|Sphider|Xenu|HTTrack|Wget|Curl|Apache Nutch|Larbin|WebSPHINX|WebCopier|TeleportPro|Offline Explorer|SiteSucker|BlackWidow|DomainCrawler|AspiegelBot|LivelapBot|spbot|DnyzBot|GPTBot|ChatGPT-User|cohere-ai|Google-Extended|omgilibot|anthropic-ai|ClaudeBot|Claude-Web) 您还可以添加其他规则以提升面网站的安全，以下是常见的过滤规则。 关键词过滤 1 (WPScan|HTTrack|antSword|harvest|audit|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf| SF/) 一句话*屏蔽的关键字*过滤 2 (?:define|eval|file_get_contents|include|require_once|shell_exec|phpinfo|system|passthru|chr|char|preg_w+|execute|echo|print|print_r|var_dump|(fp)open|alert|showmodaldialog|file_put_contents|fopen|urldecode|scandir)( 一句话*屏蔽的关键字*过滤 3 $_(GET|post|cookie|files|session|env|phplib|GLOBALS|SERVER) SQL 注入过滤 2 selects+.+(from|limit)s+ SQL 注入过滤 3 (?:(union(.*?)select)) SQL 注入过滤 6 benchmark((.*),(.*)) SQL 注入过滤 7 (?:fromW+information_schemaW) SQL 注入过滤 8 (?:(?:current_)user|database|schema|connection_id)s*( SQL 报错注入过滤 01 (extractvalue(|concat(0x|user()|substring(|count(*)|substring(hex(|updatexml() SQL 报错注入过滤 02 (@@version|load_file(|NAME_CONST(|exp(~|floor(rand(|geometrycollection(|multipoint(|polygon(|multipolygon(|linestring(|multilinestring() SQL 注入过滤 10 (substr() SQL 注入过滤 1 (ORD(|MID(|IFNULL(|CAST(|CHAR)) SQL 注入过滤 1 (EXISTS(|SELECT#|(SELECT) 菜刀流量过滤 (array_map(“ass) SQL 报错注入过滤 01 (bin(|ascii(|benchmark(|concat_ws(|group_concat(|strcmp(|left(|datadir(|greatest()

赞(0)villain2025-01-18技术笔记 阅读(171)去评论