GPC Version 2: Phoenix Rising

December 31st, 2010

I wanted to write this post right after the submission deadline earlier last week, but I put it off due to a case of not feeling well (technical term). So instead, this shall be my last piece for the year! Hopefully I’ll be writing a little more frequently in the following months, with university and a whole bunch of projects coming up. :)

Background

Phoenix Rising is a JavaScript/HTML5-based puzzle game that I wrote for the second iteration of the Game Prototype Challenge, a week-long game jam. Organized by former colleague Jason Kaplan, the purpose of the challenge was to motivate game developers. And motivate me it did; I saw it as a good opportunity to try my hand at game development. :D

Idea

This was my first ever attempt at writing a game, so I wanted to keep things simple. Basically, this meant that I couldn’t do anything that required complex mechanics. I was also working that week and had a couple of events to attend so time was definitely an issue.

I chose to write the game in JavaScript, making use of the canvas element introduced in the HTML5 specifications. There are a couple of reasons for this. One: I’m already quite experienced when it comes to web development so I wanted to take full advantage of that knowledge. Two: I haven’t actually worked with canvas before even though I’ve seen all sorts of cool demos that others have written. This was a good way for me to get my feet wet on both fronts.

It took me a few hours to come up with a workable concept once the themes for the challenge (immortality and thin ice) were announced (I was actually up ’till 4 AM thinking while reading up on basic game development). One of the first things that came to mind was the notion of cracking tiles that would deteriorate into holes once you’ve stepped on it a couple of times; a set of such tiles would essentially be a puzzle that has to be navigated carefully. This was one of the mini-obstacles in Golden Sun, a RPG that I hold in the highest regards. In the end, it turns out that I would do something similar – a set of ice tiles that melt into a trail of impassible water. I wasn’t entirely sure how to fit immortality in at first (early ideas included obtaining an elixir of life, spiritual ascendancy, etc.), but then it hit me. Why not do something featuring one of my favourite creatures of all time, the phoenix? Everything came into place once I wrapped my head around that thought, including the melting ice (a phoenix is made of fire after all), the collection of twigs to build a nest, the mini-story…

Development

Although I fleshed out the game concept fairly early in the week, I didn’t actually start coding seriously until Thursday. In hindsight I should have started sooner, but there was a lot of material for me to read up first (as stated, I haven’t done game development before). JavaScript is pretty quirky as a language too which didn’t help much (I typically use jQuery syntax for the things that I need in the average web development project).

I started with a simple game loop using setInterval() that is first called when the DOM is ready. For this particular project, a continuous game loop was probably overkill; something based on player input would have been far more efficient. I wanted the flexibility though, in case I felt like implementing AI for enemies and whatnot. It turns out this was a rather bad decision as I overestimated the performance of JavaScript in certain browsers; FF 3.6 was quite laggy although FF 4 and Chrome 8 ran the game alright.

All of the level data is stored in separate XML files which are parsed into game objects as needed. This allowed me the flexibility of adding new levels on-the-fly as well as giving an easy way for others to create their own custom maps (in the future?). The jQuery framework makes the parsing process painless, only requiring an AJAX request and a callback function. I’m actually pretty sure it’s standard practice to store this data separately, although it was more out of intuition for me.

It turns out that drawing on the HTML5 canvas is rather simple, though you are more-or-less limited to primitive objects. I made extensive use of drawImage() since most of what I had was stored externally. The visual part of the game consists of a bunch of images hastily drawn using my trusty Wacom Bamboo tablet. ;) It was actually slightly challenging drawing things that would tile correctly (mainly the wall) and writing code to display the right image wasn’t a walk in the park either. I probably messed up here; there was likely a much better way of doing things (i.e. not having separate cases for everything).

Everything else is rather straightforward; I used a 2-dimensional array to store the current game state and had various checks in place for “special” events. Arkayle’s abilities were actually a last minute add-on that I felt added a lot more depth to the game (I knew something similar had to be added eventually – there is only so much you can do with a puzzle where you couldn’t normally retrace your steps). Unfortunately since I was crunched for time, I couldn’t really do too much level design. The result is a three-level game where only the last level is slightly challenging. :P

Lessons Learned

There were a few major issues that I ran into. I found that drawing on the canvas is pretty damn slow especially when the dimensions are large (e.g. 1000px x 500px). Especially for a game like this, it was probably a much better idea to redraw when something actually happened (e.g. player input), rather than at a constant interval. An alternative would have been to not use canvas: DOM manipulation is another path that I could have taken.

A second major issue that I found was when I was trying to import my “art” assets into the game. There is an issue with drawImage() which tries to render an image onto the canvas even if it hasn’t been loaded yet. I’m not sure if this is a bug or by design (if it is by design, I’d like to know why). One of the ways to resolve this issue was to use drawImage() on image load:

1
2
3
4
5
var img = new Image();
img.onLoad = function() {
    canvasContext.drawImage(img, xStartingCoordinates, yStartingCoordinates);
}
img.src = "path/to/image.png";

Unfortunately, this caused a whole bunch of flickering due to my game loop (at least in Firefox 3.6) so I opted to preload all of my images instead.

1
2
3
4
5
6
7
8
$(document).ready(function() {
    var images = ["image1.png", "image2.png", ...];
    var img = [];
    for (var i = 0; i < images.length; i++) {
        img[i] = new Image();
        img[i].src = images[i];
    }
});

Turns out that this works quite well. :) The smart way of dealing with so many images (20 at my last count) would have been to use sprite maps to minimize the number of HTTP requests, but I didn’t want to deal with the hassle at the time (as mentioned, crunch period!).

There were a few more kinks along the way (e.g. dealing with JavaScript’s prototypical – rather than classical – inheritance), but they were pretty minor. My code base is pretty messy, though that was to be expected writing something I didn’t know how to write, using tools that I didn’t know how to use. :P In hindsight, I should have introduced a lot more separation between the engine and game logic. I also should have found a way to reuse code more effectively (see the Entity object). As for the game itself, I should have added in a few more abilities for Arkayle to use, as well as more levels (right now it doesn’t seem much like a game). Ah well…

All in all, I had a great time working on this prototype. I felt like I learned a lot from the process and I hope to apply that in future projects. :D

Torchlight

November 30th, 2010

It’s been a long time since I’ve last touched Diablo I and II, long considered revolutionary games for the action-based RPG genre. The series caused massive waves in the world of gaming, and even today we can still see its influence in modern game design. Hell, it did more than just influence the industry; the formula was so successful that a slew of copycats followed – hence the creation of the term “Diablo clone”. That’s not a bad thing, however, as refinements and innovations were made upon the basic concept.

I picked up Torchlight, among others, over the last week during Steam’s Thanksgiving sale. Being that I’ve already heard good things about the game from others, it was pretty much a no-brainer. Disappointment was not an emotion I had that day.

As I so subtly hinted in the opening paragraph, Torchlight very much feels like Diablo gameplay-wise. The similarities are such that I’d advise anyone who has played Diablo to start Torchlight at the hard or very hard setting if they want a good challenge; a lot of basic premises like elemental resistances and such are still in effect. Of course, it feels a little unfair to base my opinions of one game on another but…

Anyway, let’s start with the graphics. I definitely enjoyed the art style; it gave the game a more cartoony, light-hearted feel to it. Your mileage may vary, of course, but I find a lot of dungeon crawlers to be rather dark and dreary. It’s rather nice to see a change every once in a while.

I can’t really comment on much of the audio; I’d say it’s on par for the game’s theme. Nothing really stuck out and impressed me, but of course the game would be much worse off without it. It’s just like good web design: audio plays an invisible but vitally important role. For the most part though, I just couldn’t pay attention to most of the stuff I was listening to since I was too busy running around like a madman trying not to get killed. :P In that aspect, I suppose it did its job well, giving clear signals on when to dodge projectiles and such.

Personally I thought the story was rather weak and shallow, merely present to provide a small backdrop to the world. I barely know anything about the characters in Torchlight such as Syl and Aldric, though I guess the journals did provide some information on the latter. Hell, I don’t really know much about Ordrak either, beyond the fact that he’s a corrupt being I have to take down. Ah well, I didn’t go in expecting much anyway. :P

Gameplay-wise, Torchlight is quite solid. I wish there was a little more information on how skill damage is calculated (things like ricochet which is based on “weapon DPS”) but that’s not a really big deal (unless you’re a min-maxer, of course). There weren’t really any skill trees or synergies – just abilities you can put points into once you reach a certain level. This removed a lot of complexity in choosing how to build a character (though at the same time reducing the game’s depth). I prefer it this way though, since it lets you experiment without investing too heavily in one particular style. The pet that always tags along with you is quite a nice little innovation to the genre since it does away with the whole go-back-to-town-to-sell-everything-since-you-have-no-more-room-in-your-pack problem; the ability-granting fish was rather cool too, though quite odd and out-of-place.

One major issue I found with the game was the huge ramp in difficulty between floor 29 and 30. The Dark Zealots on floors 30-34 are quite insane as they can literally kill you in one hit, giving you next to no notice. It was quite frustrating dealing with them; indeed, I just suicided my way to the final boss because I was fed up part way through. At least in the rest of the game the difficulty ramped up at a reasonable pace; sure, there were always things that could 2/3-hit you, but you could avoid them by playing carefully. The final boss was rather tedious as well, though I was quite underleveled at that point (my character was only at 25 compared to the boss and his minions all at 30). Basically I just suicided a couple dozen times spamming traps everywhere to take him down. Good thing I wasn’t playing on hardcore mode, eh? ;)

All that said though, I believe modability is definitely Torchlight’s greatest strength. Any part of the game that causes irritation can be changed to better suit the player. For example, I thought having to constantly identify items was a massive waste of time so I hunted down something to change it. Mods can even add huge amounts of extra content to the game, from things like quests to completely new classes. There are quite a few resources for mods as well. In addition to a few select sites, TorchLeech, an unofficial tool that manages modifications to Torchlight, helps in organizing and finding most of what you need.

Anyway, in summary: definitely try this game out if you’re into action-based RPGs (hell, even if you’re only tangentially acquainted with the genre). I think there’s a free demo floating about somewhere too. :P

StarCraft II: Lost Viking

October 31st, 2010

For anyone not accustomed to side-scrolling shooters – very plausible given that SC2 is a real-time strategy game – the Lost Viking achievements may be one of the most annoying to obtain (at least as far as the campaign goes). However, given a bit of effort, it is far from impossible for even the most casual gamer. Here are a few general tips…

Power-ups

  • Plasma vs. side missiles – Though the plasmas will take down bosses much quicker than the rockets, it is hard to use them elsewhere in the scenario. In general, the latter is much more user-friendly; the former is for those who have great skill in aiming while maneuvering through enemy fire.
  • Drones – These little guys are amazing; they help take down opposing units and also sacrifice themselves should you take a hit, acting as an extra life. Two of these should be gotten as soon as possible (wait for the power-ups to morph if need be).
  • Bombs – Bombs are literally the PANIC! button in this minigame and there is no shortage of them after having finished upgrading the ship’s weaponry and obtaining the drones. They should be used whenever you’re in a “crap, I’m screwed”-spot (typically during the Zerg and Terran bosses).

Levels

  • The Protoss level is pretty straightforward; there aren’t too many things worth mentioning about it. One interesting thing to note is that you can “farm” the interceptors that the boss throws out for points; this can actually get you past the 500k point barrier without having to fight Terra-Tron for a 3rd time.
  • For the Zerg, it is best not to move around a lot as you take down the scourges. By staying motionless, the enemy “firing” pattern is much less chaotic. Of course, moving as little as possible is a general rule in most shooters so… As for the boss here, the main thing to watch out for is the tentacles; the stockpile of bombs will come in useful here though they aren’t completely necessary.
  • Terra-Tron is by far the biggest threat in the third level, with lasers, bullets, and even a homing saw at its disposal. The laser can be avoided by being at the opposite side of the screen from where Terra-Tron starts firing; the saw isn’t too hard to dodge, at least on its own. The hail of bullets on the other hand is quite annoying given the clunky, imprecise controls. Again, the bomb stockpile will serve quite nicely here.

Controls

One interesting thing to note is that spamming the spacebar continuously is much more effective than keeping it held down. This gets rather tiring and distracting so, me being me, I wrote a quick AutoHotkey script to automate this process.

1
2
3
4
5
6
7
8
9
10
11
Activated := 0
 
Loop {
	If (Activated) {
		Send {Space}
		Sleep, 50
	}
}
 
$Space::
	Activated := !Activated

Press the spacebar once and the ship will fire continuously at maximum speed without any further direction. It makes it much easier to focus on dodging fire and using bombs when necessary. :P

Anyway, Happy Hallowe’en!

Microsoft Kinect

September 30th, 2010

It’s been over a year now since Project Natal (now Kinect) was announced. One of the most mindblowing demos of the day was created by Lionhead, where participants could interact fluidly with an in-game kid named Milo (E3 2009 video here). Things sort of died down after a while until earlier this year when Microsoft formally introduced the system as Kinect. Curious name, that; it’s sort of a blend between kinect (i.e. movement) and connect (as in with other people) which is understandable given all the hype about social media these days.

Anyway, my friend and I got to play around with the system earlier today (quite randomly, actually, since we were just strolling downtown getting food and stuff). We got to play Kinect Adventures which included activities such as navigating a raft with our bodies, dodging items on an obstacle track, and plugging up leaks caused by some very vicious fish. I believe the second game we tried was Kinect Joy Ride which was basically a racing game with a variety of game modes. I was actually quite surprised at how well the sensor picked up body movements, though I did have some problems controlling my car in Joy Ride (though that was likely due to my inexperience with the controls). Both games had relatively slick UIs too given the new method of control; I can’t think of many ways to improve upon them off the top of my head.

Perhaps the technology is still too immature or the style of games will only fit in with a niche group of people; nevertheless, it’ll be interesting to see how Microsoft, Sony (with its PlayStation Move), and Nintendo (with its Wii – or presumably, the Wii’s successor) battle it out for a piece of console gaming’s future.

StarCraft II

August 21st, 2010

Well, SC2 is a game that needs no introduction with all the press that it has gathered over the last few months. After all, its predecessor literally jumpstarted the entire progaming scene; expectations are high that SC2 will take Brood War’s place over the next few years. I’ve been replaying the campaign the last few days, trying to pick up most of the achievements (I already stomped through it once during my exam period just to go through the storyline; let’s just say casual difficulty is really, really casual :P ).

As is usual for a Blizzard game, SC2 mostly lives up to its hype. The graphics are phenomenal without being overbearing; I can still play the game relatively well on medium settings using a laptop that I got three years ago. The audio is pretty decent as well. All of the sounds are appropriate, given the situation; it’s easy enough to tell what’s going on if you listen intently. Much of the music seems like it was remixed from SC1 too, giving an old nostalgic feel to the game.

What I really liked was the changes to the user interface. The new control groups really help a low-APM player like me. :P Things like automatically (and intelligently) choosing which building to build a unit from, having unlimited units in one group, casting only one copy of a special ability at a time, and such means that I can focus less on mass-spamming my mouse and keyboard and focus more on overall strategies. Of course, there are things missing that I’d like to see. For example, wouldn’t it be nice if units created from a building were automatically assigned to a chosen control group when they come out? This could be implemented by, say, Ctrl + Right Click on a control group when the buildings are highlighted. Right now this is half-done if you rally the buildings’ troops to a unit; they’ll basically follow that unit if it isn’t dead.

The storyline itself was okay. I didn’t expect much from a RTS game so all of the character development and stuff were pretty much gravy. That said, much of the plot is relatively predictable and cliché; despite giving you a lot of choice in choosing which missions to do first, there isn’t much overall impact in the end. At least there weren’t any inconsistencies that I could see though (e.g. one mission done before another, but some dialogue refers to the latter in the former). I suppose not too many people care about singleplayer SC2 anyway so it’s a moot point…

I haven’t played any league games yet and don’t really intend to; I’m not particularly big on competitive gaming in general. SC2 isn’t exactly balanced for anything above 1 vs 1 anyway (hell, it’s not even balanced for that despite aiming to become the de facto standard for progaming). I’ll stick with trolling the AI and playing custom maps, thanks. :P

Anyway, there are a few things that I dislike about the game. The lack of chat on the new Battle.net is kind of a step back from the previous iteration; not everyone likes to interact with others on the official community forums. That said, the old Battle.net was pretty blah, so meh… Regionalization of the community is a massive pain in the ass too; is Blizzard seriously telling me that I can’t play with friends in other regions without buying another copy of the game? The world is coming closer together with all of our new communication technology and they pull that kind of crap in the name of providing a better experience for their users. I haven’t seen a justification for such an action that couldn’t be met with a “softer” version of segregation (for example, league matchmaking can be done by region but custom games can allow anyone from anywhere to join; this allows for lower ping in “important” matches and increased versatility when players want it). Sure, it’d be harder to implement but it shouldn’t be impossible for a company with Blizzard’s resources. Oh yeah, a lack of LAN support is lame too (I can deal with regionalization if we had that option). I can definitely see why Blizzard doesn’t want to implement it though, seeing as SC2 is supposed to be focusing on competitive gaming. A look at the company’s actions with KeSPA and you can see that they really want to control a good part of the scene.

That said, SC2 is still a solid game in my eyes (and I’m missing out on the supposed best part of the game – online league play). Time to finish up the campaign achievements, I guess. My fingers are still numb from doing “All In” on brutal versus all of that flying Zerg… :P

RuneScape Bonus XP Weekend: An Overview

March 21st, 2010

On February 26th, Mod Knox announced RuneScape’s first ever bonus XP event. It was to run on the weekend of March 12th, starting at 7:00 AM EST and running until the same time the following Monday.

The effects of this announcement were profound and widespread. In a matter of days, many popular items used in skill training (such as planks, herbs and summoning secondaries) became unbuyable on the Grand Exchange. Merchanting clans – and indeed, pretty much everybody – promptly jumped on the bandwagon, causing prices to spike in a dramatic fashion. This imbalance in the supply and demand of raw materials trickled down; soon, “essential” items such as potions were also bought out, crippling the ability of players to participate in PvP and other activities.

In a bid to hoard more supplies, however, many players began to sell off expensive items such as godswords and Bandos armour. This triggered a crash of sorts in the high-end market, causing many players to speculate on what the future has in store. What was notable, however, was the influx of discontinued items in the Grand Exchange; prior to the event announcement, traders often had to use “junk” in order to obtain them. There are two sides to this: although collectors were easily able to get something they want at a steep discount, said items became more heavily concentrated in the hands of wealthy players.

This state of affairs wouldn’t last for long, of course. About a week later, the market for training supplies began to cool down; it won’t be until the beginning of the actual event that things start to pick up again. Prices for high-end items took on a reversed course as people began to recognize that they were vastly undervalued.

Meanwhile, the RuneScape forums were raging with activity. Many players thought it was an awesome event and began preparing for it, even going so far as to use calculus to determine how much XP might be expected for a particular activity. Questions were thrown around left, right and center about what particular training methods were covered under the event. Others, however, had a negativistic view and stated that bonus XP devalued skills (in particular, summoning).

On the first day of the actual event, people quickly determined that the XP rate counter was decreasing faster than it should have been. Mod Nexus confirmed this in a forum thread, saying it was because of a rounding error in the math calculations. This meant that the curve leveled off at 110% normal experience approximately 3 hours faster than expected, throwing off many of the more meticulous plans for the day.

Despite this miscalculation, players were undeterred in their training. Areas such as Castle Wars and the Ourania Runecrafting Altar were flooded with skillers. At multiple points throughout the weekend, RuneScape peaked at over 180k people online. Such enthusiasm turned what is ordinarily dull training into a group activity, building relationships between players; it was sort of like a global skilling clan going to areas normally devoid of people. In the end, many ‘Scapers received a hefty experience boost in the skill(s) of their choice.

So that’s pretty much what happened. Here’s a few of my thoughts on the event.

There’s no doubt that it was one of the most controversial announcements of the year. Although bonus XP events were commonplace in many other MMORPGs, this marks the first time such a thing has happened in RuneScape. Much of the debate centered around the “devaluing” of achievements prior to the weekend. In particular, summoning took the brunt of the criticism due to the design of the skill itself; most of the time spent is on charm gathering rather than pouch making, so it was possible to save months of playtime (rather than the ~5 or so hours with most other skills). Herblore took second place due to the vast amounts of money that could be saved in the process of gaining upwards of ~4 million XP over the weekend.

What’s the big deal? Well, RuneScape is largely a time-based game (more so than most); there are few things that players can’t do if they put in sufficient effort. As a result, the main thing distinguishing players is the number of hours spent in-game (way overgeneralizing here). To put it into perspective: think of time as some sort of currency and a government deciding to introduce more of it into the economy – causing hyperinflation to occur.

It’s essentially an egotistical argument; the idea boils down to “I worked for my skills the hard way, so everyone else should too”. In a game like RuneScape, where higher levels give substantially larger advantages, it becomes easy for players to point fingers at the bonus XP event and say that it nullifies their competitive advantage over others. For example, let’s use the pack yak as an example: it requires level 96 summoning and provides a massive advantage in the form of extra inventory space (and long-range banking). Those who have gotten the yak prior to the event were able to cash in on their efforts, staying at boss fights and whatnot longer than their lower-leveled counterparts. However, there may be those who have, say, level 85-90 summoning who didn’t have enough resources to reach level 96, but had enough to achieve half of that. The bonus weekend helps to put the pack yak in their line of sight, making it possible for more people to compete with others who have already done so.

Now, I’m a relatively casual player. Although I’ve been “around” longer than probably 95% of the population today, I haven’t done many noteworthy things. For example, I’ve never hit maxed level in any skill thus far, and have no plans on doing so. My reason? I dislike grinding and am happy with ~90% of the benefits in a third of the time. Basically, my opportunity cost for gaining the last little bit of extra advantages are too high. And yeah, being a mathie I have a habit of calculating efficiency rates for various activities in the game. :P So what draws me into RuneScape? Questing is probably the primary reason. It’s also one of the few games that I’m capable of multitasking well with when I’m knocking heads together in Guild Wars or programming “the next big thing”. ;) For someone like me who pretty much tries everything to avoid straight-up grind, anything that increases efficiency per time spent is likely to be more good than bad (just as a sidenote, I went for summoning, construction and hunter last weekend).

There’s a couple of arguments in favour of the event that I haven’t really seen yet. First of all, it was sort of a money-sink. In the case of summoning, many players had to buy shards from the shop, removing gold from the game. It’s no big secret that inflation has been running rampant over the last couple years so every little bit helps. For other materials, it helped to decentralize wealth as people with a lot of money bought things from those who supply them (merchants notwithstanding). Secondly, an increased number of higher-leveled players can actually help the game. Think of it this way: if more people are able to make an attempt at the God Wars Dungeon or the Corporeal Beast, then prices of the more expensive items may drop into affordable range.

It’s been my experience that forums aren’t a very good way to judge feedback to an in-game event. The responses tend to be skewed towards the dissenting side and for good reason – happy players tend to be the ones actually playing the content. I suspect given the amount of activity last weekend that it was a huge success, so I wouldn’t be surprised if another similar announcement comes up in the future.

I appreciate a gaming company who’s willing to try out controversial things so I have a few suggestions for Jagex. It’d be nice to give less notice should a repeat occur, as the Grand Exchange was essentially destabilized for a month. Using a limited subset of activities might be interesting too, as that would herd players to various areas (this, of course, only works if the activity isn’t based around zero-sum like mining).

Anyway, this post has gotten a bit out of hand so I’ll wrap things up for now. :P

Guild Wars Community Event: Maragon

March 6th, 2010

Last week the KISS alliance hosted its annual Maragon on Guild Wars. Basically it’s a massive riddle-solving and running event spanning over two long days.

I haven’t even been on Guild Wars for a significant period of time since last year, so this was kind of interesting. In the beginning I thought I’d be at a severe disadvantage due to the time lapse; at least I’d have Dacelo et. al. backing me up though. ;)

In hindsight I didn’t have much to worry about. The group riddles weren’t all that hard – although there were a few “wtf” moments. I mean, how did we manage to derive Sifhalla as the endpoint three times consecutively? :P Some of the runs were definitely crazy though: the two runs in the Realm of Torment and the HM run from Arbor Bay to Gadd’s Encampment.

Good thing as a “PUG” I had the advantage of running in full armour. :P The change to Shadow Form last Thursday and its subsequent unbanment (I officially declare unbanment to be a real word) from the event was a godsend as well. I don’t know if it’d be possible to solo run half of those routes otherwise.

Anyway, some pictures:

Maragon - Deldrimor War Camp

Taking a nice long break after the crazy run from Beacon's Perch to Deldrimor War Camp

Starting the solo riddle for Prophecies.

Crowding around the leader of KISS - James - before the start of the final solo riddle of the event.

All in all it was good fun. I definitely got a lot of running practice over that weekend. :P Now if only I remembered to record one of the more challenging runs on video… And congratulations to Dacelo for winning first prize (Obsidian Armour set) in the Nightfall portion of the event. It was totally worth participating just to hear Tessa’s expression. ;)

I think my brain has fully recovered now.

Random Thought of the Day

February 8th, 2010

Ever had this crazy awesome vision in your head but just can’t get it down into a physical format? It could be an idea for a project, a solution to an issue, whatever. Well, that’s basically the sort of problem I’ve been dealing with for the past few weeks or so.

The problem: create a design for the front page of my site (and perhaps this blog too, in time). The vision: a dark theme, using fantasy and sci-fi material as inspiration (planets, stars, floating islands…). The solution: uhh…?

I call it “creative paralysis”. It’s something similar to writer’s block (which, incidentally, I happen to encounter constantly when I’m writing academic papers and such). I suppose it’s different in that the idea is already at hand; the issue is figuring out what to do with it.

For me, I’m thinking that my lack of experience in Photoshop (and artistic skills in general) is holding me back. I’ll probably end up messing around randomly and hope things work out. Who knows, I might actually learn something in the process…

I probably have a few hundred or so images in my head that I’d love to get on paper. Hell, my lucid dreams can probably form the basis of several short stories. Now, if only the details were a bit clearer… :P

Seriously, if there was a machine that could materialize thoughts into physical matter, I’d venture to say that we’d be living in a rather interesting world. ;) In the meantime, I’d be happy if virtual/augmented reality stuff went mainstream.

Building a Web Scraper Using JavaScript and jQuery

January 25th, 2010

When building web applications, sometimes there’s a need to fetch data from other sources. Perhaps you’re building a custom RSS feed of news items based on your interests or you want to aggregate data from several sites. In any case, it’s not always possible to do this elegantly; you may not have direct access to the raw data and an existing API may not exist. For these situations there’s one general (albeit fragile) solution: manually parse the end result when a page is loaded in a browser.

There are many different ways to build a web scraper. A server-side language such as PHP will have a much easier time as there are less limitations and existing libraries (such as cURL) to accomplish the task. I’ll be using JavaScript (with the jQuery library), however, as I want a standalone client independent of server technology.

Let’s start with a random snippet of code:

1
2
3
4
5
6
7
8
9
10
11
12
function fetchPage(url) {
    $.ajax({
        type: "GET",
        url: url,
        error: function(request, status) {
            alert('Error fetching ' + url);
        },
        success: function(data) {
            parse(data.responseText);
        }
    });
}

The above function attempts to make an AJAX call to a specified web page, fetch the HTML of said page and pass it to a parsing function. Simple enough, right?

Unfortunately there’s a little problem with this. Client-side programming languages have to deal with something called the same origin policy which basically restricts scripts from accessing external domains. From a security standpoint this is obviously a good thing; however it can be a headache for web app. developers (indeed, there are ideas such as the origin HTTP header to solve this). (The following bit can be ignored if the scraper is on the same domain as the data source – but in such a case why is said scraper even necessary?)

In this case, there are a couple of solutions (that I can think of). Both of them take advantage of the fact that JavaScript can read JSON even if it is located on a different domain.

  • The first one is writing a complementary server-side script that takes a few arguments (such as the target URL), makes the call, parses the result into JSON format and passes it back to the calling function. It’s a simple idea, but I never really liked it because it introduced a major dependency (which, in my case, can be a pain to deal with). However, it’s definitely a viable solution which works extremely well.
  • The second is using an existing system such as Yahoo’s YQL to fetch the required data and return it in a structured form. This is the method that I’ll be using.

YQL is an interesting little beast. From their site:

The Yahoo! Query Language is an expressive SQL-like language that lets you query, filter, and join data across Web services. With YQL, apps run faster with fewer lines of code and a smaller network footprint.

I haven’t had too much time to look into it so I do what all programmers do: Google the living crap out of a problem to find a solution. :P In this case, Chris Heilmann has a great post on how to load external content via. various methods including YQL. To simplify things, James Padolsey wrote a plugin that detects an external AJAX call and passes it to YQL automatically.

Anyway, that solves the problem of fetching data from an external source. All that’s left is extracting the relevant pieces for whatever application is being built. Consider the following example:

1
2
3
function parse(data) {
    alert($(data).find("h1").text());
}

The parse() function takes the responseText data from the earlier fetchPage() function and then proceeds to pick at it slowly and painfully. What’s really cool about it is that pretty much all of jQuery’s selectors can be used to select relevant data. In the above case, I’m trying to extract text inside the first <h1> tag found on a page and outputting it as an alert to the browser. Obviously there are more complex uses for this but they are outside the scope of this post. :P

And…that’s it! :D Those two code blocks combined pretty much handles all of the grunt-work in extracting data from external sources. Again, this is a rather fragile solution – it can break if the target page’s HTML changes (one possible solution is to pick unique identifiers or classes). In addition, some sites may have restrictions on their data – but I’m sure everyone reads those long-winded documents. ;)

“What’s Your Fastest Text Input Tool?”

January 24th, 2010

Here’s an interesting post on Lifehacker: What’s Your Fastest Text Input Tool?.

I’ve been using full-sized keyboards for a *very* long time now so that blows everything else out of the water for me. When I’m motivated I can type at speeds exceeding 100 words per minute, although that’s typically not necessary with the stuff I do.

I’d rank pen and paper as second. It’s pretty much a necessity in university when you’re taking down notes and whatnot. Although some people can get away with using laptops in class, I find that it’s extremely hard to copy out diagrams and mathematical equations. Perhaps this is some indication that I should learn LaTeX or something? :P

Next up would probably be a close race between my Wacom tablet (using Window’s handwriting recognition software) and my iPod Touch. I’m still rather slow at both, although that might just be because of a lack of practice.

In last place is a standard phone with a three-letters-per-number keypad. Yeah. I don’t use my phone very often and I text even less. :P