Corey Salzano

Category: Robots

Web crawling robots are pieces of software that copy and manipulate the data available on the world wide web. I run many websites, and therefore have a love/hate relationship with the robots that reach out to me each day.

  • Allow robots to crawl your wp-content folder

    An alternate title for this post could be, “How disallowing robots from your wp-content folder could cost you mobile rankings in Google.”

    On April 21st, 2015, Google is going to change the way it ranks sites for users on mobile devices. By blocking Googlebot from your plugins folder, you could be preventing Google from deciding that your site is mobile-friendly. If you are skeptical about this Google-is-changing statement I have made or want to dive into the details, read this.

    So, why?

    Why does Google need to crawl your plugins folder? Plugins often contain CSS or JS files, and those files are necessary to understand what the page actually looks like. Google Webmaster tools told me I was preventing Googlebot from crawling some CSS files in which it was interested. Robots need to download all CSS and JavaScript files or they cannot determine if a page is friendly to mobile users.

    I found this line in my client’s robots.txt:
    Disallow: /wp-content/plugins/

    Why would this line be in robots.txt at all? My client lives on GoDaddy Managed WordPress Hosting, and that service creates a robots.txt file that looks like this (as of the date I published this post):

    User-agent: *
    Crawl-delay: 1
    Disallow: /wp-content/plugins/
    Disallow: /wp-admin/

    There are a bunch of blogs that discuss the “ideal WordPress robots.txt file” that recommend blocking the plugins folder, and some plugins alter robots.txt to block this directory, too. Before February 2015, even Yoast SEO did this. It’s no longer a good idea.

  • How to Block Java user-agents

    A variety of user-agents that begin with “Java” are likely visiting your website. Visits providing this type of user-agent are programs created in Java by developers who did not choose to change the default user-agent string value. Here is a list of the Java user-agents I have encountered:


    Java/1.4.1_04
    Java/1.5.0_02
    Java/1.5.0_06
    Java/1.5.0_14
    Java/1.6.0_02
    Java/1.6.0_03
    Java/1.6.0_04
    Java/1.6.0_07
    Java/1.6.0_11
    Java/1.6.0_12
    Java/1.6.0-oem

    I will maintain this list simply for kicks. There is no need to collect an exhaustive list of these user-agent strings in order to block them. As I have mentioned before, I prefer to ban non-human visitors based on a combination of an IP address and a user-agent string.

    URL rewrite rules

    Here are some URL rewriting conditions and rules that will match a list of IP addresses and any user-agent that begins with “Java” and deliver a 403 Forbidden response for any HTTP request to your server:


    RewriteCond %{HTTP_USER_AGENT} Java.*
    RewriteRule ^/(.*)$ /$1 [F]

    The condition matches any user-agent string that begins with “Java” no matter what comes later. The rewrite rule returns any location that was requested with a 403 Forbidden response code. There will be no change made to the URL and no document delivered.

    IIS7 URL Rewrite web.config

    
<rule name="no-java-bots" stopProcessing="true">
    <match url="(.*)" />
    <conditions>
	<add input="{HTTP_USER_AGENT}" pattern="^Java/.*" />
    </conditions>
    <action type="AbortRequest" />
</rule>

    Why block Java bots?

    Bots with a well-defined purpose will typically identify themselves with a unique name. These Java user-agents are either not interested in identifying their purpose or not ready to publish their name and take ownership of the crawling activities. Both cases are a waste of bandwidth. Test your new application on someone else’s website. Play with your shady crawler on someone else’s website. Come back when you are willing to identify yourself.