block yandex bot

yandex bot  yes Yandex Search engine crawler . stay  yandex bot user agent  In this article , I summed up all  yandexbot  Of User-agent, Through that article , We can see some Yandexbot comply with  robots.txt agreement  , Some do not comply robots.txt agreement .

We need to block yandexbot（block yandex bot） Can pass robots.txt and IP The way to shield , Let's talk about it separately .

adopt robots.txt shielding yandexbot

yandex bot user agent  In this article, only abide by robots.txt The crawler of the agreement, we can pass robots.txt Protocol to shield , If we don't comply, we can't pass robots.txt Protocol to shield .

for example , We need to block YandexBot/3.0 This Reptiles ,robots.txt It is written as follows ：

User-agent: YandexBot

Disallow: /

This prohibits all YandexBot Grab any page of our website , Blockade reached yandex bot（block yandex bot） Purpose .

If we just want some pages not to be YandexBot Grab , We can write this way ：

User-agent: Yandex

Allow: /

Disallow: /private/user-info

We allow YandexBot Visit all pages , But the last one forbids access  /private/user-info  page , That's what we're trying to achieve .

Some more Yandex bot Do not abide by robots.txt agreement , So we need to go through IP perhaps IP The address segment is shielded .

adopt IP Address to shield Yandex bot

We go through  YandexBot  The page can find out the specific Yandex bot Of IP Address , for example ：YandexMobileBot/3.0  Do not abide by robots.txt agreement , So we need to pass shielding IP The way to shield Yandex bot, Displayed through this page IP , Let's choose any one IP As an example ：95.108.213.6.

If our front-end server （ for example ：Nginx,Apache） Running Linux We can directly enter the following command to mask Yandex bot：

sudo iptables -A INPUT -s 95.108.213.6 -j DROP

such YandexMobileBot/3.0 We can't capture the content on our website .

We can input one by one IP Address , Shield through firewall .

summary

shielding Yandex bot We used two methods , One is ： adopt robots.txt To shield , The other is ： adopt IP Address to shield , If Yandexbot Non compliance rotbots.txt agreement , We will use IP Address to shield .

Yandex bot yes Yandex Search engine crawler , If we block , It may bring us losses in traffic , Please consider whether it is necessary to shield Yandex Bot, Make another decision .