-
Allow robots to crawl your wp-content folder
An alternate title for this post could be, “How disallowing robots from your wp-content folder could cost you mobile rankings in Google.” On April 21st, 2015, Google is going to change the way it ranks sites for users on mobile devices. By blocking Googlebot from your plugins folder, you could be preventing Google from deciding…
-
How to Block Java user-agents
A variety of user-agents that begin with “Java” are likely visiting your website. Visits providing this type of user-agent are programs created in Java by developers who did not choose to change the default user-agent string value. Here is a list of the Java user-agents I have encountered: Java/1.4.1_04 Java/1.5.0_02 Java/1.5.0_06 Java/1.5.0_14 Java/1.6.0_02 Java/1.6.0_03 Java/1.6.0_04…
-
Blacklisting via Ionic’s Isapi Rewrite Filter
In IIS, banning IP addresses from accessing a website is fairly easy. I rarely do this, however, because I prefer to use a combination of an IP address and a user agent string to identify bad bots that are likely scraping my content or attempting to harvest email addresses. I try to avoid blocking an…