In case Googlebots creates a large amount of connections to the website, if your server has a Mod Security or CSF Firewall installed, then Googlebots may be misunderstood as attacking the website and will be blocked. 
Consequently, Googlebots will no longer be able to access your site. To fix this, we need to adjust the Mod Security Rules or CSF Firewall configuration as follows: 1. 
With Mod Security Rules

1 #Allow googlebots 
2 SecRule REMOTE_HOST $ allow, pass 

4 #Add this to your config. The right way to identify Google bot is it’s User-Agent. 
5 Secrule REQUEST_HEADERS: User-Agent “Googlebot” phase: 1, nolog, allow, ctl: ruleEngine = off 

7 # GoogleBot by user-agent … 
8 SecRule HTTP_USER_AGENT “Google” nolog, allow 
9 SecRule HTTP_USER_AGENT “Googlebot” nolog, allow 
10 Secule HTTP_USER_AGENT “GoogleBot” nolog, allow 
11 Secrule HTTP_USER_AGENT “googlebot” nolog, allow 
12 Secrule HTTP_USER_AGENT “Googlebot-Image” nolog, allow 
13 ## 
14 SecRule HTTP_USER_AGENT “AdsBot-Google” nolog, allow
15 Secrule HTTP_USER_AGENT “Googlebot-Image / 1.0 ″ nolog, allow 
16 Secrule HTTP_USER_AGENT” Googlebot / 2.1 ″ nolog, allow 
17 Secrule HTTP_USER_AGENT “Googlebot / Test” nolog, allow 
18 SecRule HTTP_USER_AGENT “Mediapartners-Google / 2.1 ″ nolog, allow 
19 SecRule HTTP_USER_AGENT “Mediapartners-Google *” nolog, allow 
20 SecRule HTTP_USER_AGENT “msnbot” nolog, allow

2. With CSF Firewall
We add the following paragraph /etc/csf/csf.rignore file:

Good luck!

Read more :

Leave a Reply

Your email address will not be published. Required fields are marked *