There are many different ways to build a web scraper. A server-side language such as PHP will have a much easier time as there are less limitations and existing libraries (such as cURL) to accomplish the task. I’ll be using JavaScript (with the jQuery library), however, as I want a standalone client independent of server technology.

Let’s start with a random snippet of code:

1
2
3
4
5
6
7
8
9
10
11
12

function fetchPage(url) {
    $.ajax({
        type: "GET",
        url: url,
        error: function(request, status) {
            alert('Error fetching ' + url);
        },
        success: function(data) {
            parse(data.responseText);
        }
    });
}

The above function attempts to make an AJAX call to a specified web page, fetch the HTML of said page and pass it to a parsing function. Simple enough, right?

Unfortunately there’s a little problem with this. Client-side programming languages have to deal with something called the same origin policy which basically restricts scripts from accessing external domains. From a security standpoint this is obviously a good thing; however it can be a headache for web app. developers (indeed, there are ideas such as the origin HTTP header to solve this). (The following bit can be ignored if the scraper is on the same domain as the data source – but in such a case why is said scraper even necessary?)

In this case, there are a couple of solutions (that I can think of). Both of them take advantage of the fact that JavaScript can read JSON even if it is located on a different domain.

• The first one is writing a complementary server-side script that takes a few arguments (such as the target URL), makes the call, parses the result into JSON format and passes it back to the calling function. It’s a simple idea, but I never really liked it because it introduced a major dependency (which, in my case, can be a pain to deal with). However, it’s definitely a viable solution which works extremely well.
• The second is using an existing system such as Yahoo’s YQL to fetch the required data and return it in a structured form. This is the method that I’ll be using.

YQL is an interesting little beast. From their site:

The Yahoo! Query Language is an expressive SQL-like language that lets you query, filter, and join data across Web services. With YQL, apps run faster with fewer lines of code and a smaller network footprint.

I haven’t had too much time to look into it so I do what all programmers do: Google the living crap out of a problem to find a solution. In this case, Chris Heilmann has a great post on how to load external content via. various methods including YQL. To simplify things, James Padolsey wrote a plugin that detects an external AJAX call and passes it to YQL automatically.

Anyway, that solves the problem of fetching data from an external source. All that’s left is extracting the relevant pieces for whatever application is being built. Consider the following example:

1
2
3

function parse(data) {
    alert($(data).find("h1").text());
}

The parse() function takes the responseText data from the earlier fetchPage() function and then proceeds to pick at it slowly and painfully. What’s really cool about it is that pretty much all of jQuery’s selectors can be used to select relevant data. In the above case, I’m trying to extract text inside the first <h1> tag found on a page and outputting it as an alert to the browser. Obviously there are more complex uses for this but they are outside the scope of this post.

And…that’s it! Those two code blocks combined pretty much handles all of the grunt-work in extracting data from external sources. Again, this is a rather fragile solution – it can break if the target page’s HTML changes (one possible solution is to pick unique identifiers or classes). In addition, some sites may have restrictions on their data – but I’m sure everyone reads those long-winded documents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

function blah() {
    var pagedata;
    $.ajax({
        type: "GET",
        url: "test.php",
        error: function(request, error) {
            alert("Error: " + error);
        },
        success: function(data) {
            pagedata = data;
        }
    });
    return pagedata;
}

Looks reasonable enough. The purpose of the function is to make a GET request to some page (“test.php” in this case), store the result in a temporary variable and return said variable (of type string).

However, it turns out that this doesn’t work too well. When the function is called, chances are pretty good that the returned string is empty. Why? At first I thought it was because of some weird variable scoping rules in JavaScript, but that didn’t really make too much sense.

It turns out that I forgot about the “A” in “AJAX” (i.e. Asynchronous JavaScript and XML). The function above doesn’t execute in a top-bottom manner; the $.ajax() call is made in parallel with the rest of the script. Generally this isn’t too much of a problem (indeed, it’s a valued property by those making web applications) but in this case it isn’t what I want.

Anyway, there’s a simple solution to this. What I want to do is make the call synchronous with the rest of the script. Fortunately, jQuery makes this extremely easy in the $.ajax() method by providing an option (async: false) to turn this off. Thus:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

function blah() {
    var pagedata;
    $.ajax({
        async: false,
        type: "GET",
        url: "test.php",
        error: function(request, error) {
            alert("Error: " + error);
        },
        success: function(data) {
            pagedata = data;
        }
    });
    return pagedata;
}

This locks up the browser while the call is being made so it’s probably not the best solution. It shouldn’t matter too much with small requests though (or requests made with the user’s consent via. the UI or something).

For this to work, you must be running Apache as your web server (I suppose IIS should have a similar method, but eh…) and have mod_rewrite enabled. We’ll be creating (or adding to) a .htaccess file as follows:

1
2
3
4

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(www\.)?domain.com$
RewriteCond %{REQUEST_URI} !^/subdir/
RewriteRule ^(.*)$ /subdir/$1

The first line turns the runtime rewriting engine on to allow for modifications. The second line determines whether the request came from “www.domain.com” or “domain.com”; this is to prevent rewriting the URLs of any subdomains you might have. The condition after checks to see whether the request has already been redirected to the subdirectory. If not, then the rewriting rule is executed to make it so.

The above code is sufficient if all you want is to move files from “/www/” to a subdirectory “/www/subdir/”. However, Apache is quirky in that it requires a trailing slash at the end of directory names (to specify that it is indeed a directory). Normally this isn’t a problem since there is a module (mod_dir) that automagically redirects a path without a trailing slash to one that does. However, it conflicts with the above code in this case: first a request is directed to a subdirectory and then mod_dir issues another redirect, exposing the subdirectory name. For example, a request for “http://irythia.com/somefolderhere” will become “http://irythia.com/subdir/somefolderhere/”.

Note: mod_dir only executes if there isn’t a trailing slash on directory names, so if you qualify URLs with one the rules above work as expected. That is, “http://irythia.com/somefolderhere/” won’t change.

To fix this problem, we add another rule to append a trailing slash on all URLs ending in directory names.

1
2
3
4
5
6
7

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(www\.)?domain.com$
RewriteCond %{REQUEST_URI} !(/|\.[^/]*)$
RewriteRule (.*) http://www.domain.com/$1/ [L,R=301]
RewriteCond %{HTTP_HOST} ^(www\.)?domain.com$
RewriteCond %{REQUEST_URI} !^/subdir/
RewriteRule ^(.*)$ /subdir/$1

Note the additions in lines 2-4. Again, line 2 checks to see whether the request came from “www.domain.com” or “domain.com”. Line 3 checks to see if the URI ends in a directory by filtering out file extensions (i.e. .php, .html, .js, .css, etc.). It also checks to see if there isn’t already a trailing slash. Finally, we rewrite the URL. The [L,R=301] flag tells Apache to not execute any more rules and do a permanent redirect.

And that’s that. Upload the .htaccess file to the root web directory (/www or /public_html) and you’re good to go.

Some final notes: this is essentially a hack to work around limitations imposed by many web hosts; there are probably better methods if you have direct access to the server. Since it is a hack, it might not work on all sites without modification; this post is pretty much a reference for myself. As well, there is likely some performance penalty (as minor as it is) because the .htaccess file rewrites all requests that hit the domain. Of course, if you were that worried about performance, you wouldn’t be running on a shared host anyway.