A few days ago, around the time of my previous post, I was in Eve Online when the cry went out that there was something amiss with our alliance’s website (or guild, for those of you who play World of Warcraft). The web server was throwing up a mix of Server Error 500 pages, and half-rendered pages complaining about a missing database.
Whilst I don’t administer the server or the website, word soon got around that the database was indeed missing. The drive which held the database and its backups had become corrupted and was unrecoverable, fortunately the web server itself was sitting on a different drive. When the webserver came back online, the only recoverable backup was over 3 months old, causing a bit of a timewarp with forum posts and a whole chunk of game data missing.
To tangent for a bit, Eve Online is a single-shard space-based MMOG, very similar to Braben and Bell’s Elite. Just like Elite, pilots gain kudos and credit for defeating their enemies, and this competitiveness has driven a lot of community based projects. There are many websites which will take the computer-generated kill receipts (known as killmails), and provide a search-able database (called a killboard) for players to keep track of their progress. For reference, an example killmail is below:
2009.01.05 23:56 Victim: phree4u Corp: Shadow-Tech. Industry's Alliance: None Faction: NONE Destroyed: Caracal System: 5ZXX-K Security: 0.0 Damage Taken: 10967 Involved parties: Name: Morrganna (laid the final blow) Security: 5.0 Corp: VentureCorp Alliance: Imperial Republic Of the North Faction: NONE Ship: Cerberus Weapon: Caldari Navy Scourge Heavy Missile Damage Done: 10967 Name: BigggSexy Security: 3.3 Corp: g guild Alliance: Imperial Republic Of the North Faction: NONE Ship: Falcon Weapon: ECM - Spatial Destabilizer II Damage Done: 0 Destroyed items: Bloodclaw Light Missile, Qty: 144 Large F-S9 Regolith Shield Induction 10MN Afterburner II Ballistic Deflection Field II Heat Dissipation Field II Bloodclaw Light Missile, Qty: 230 (Cargo) Anti-Kinetic Screen Reinforcer I Anti-Thermal Screen Reinforcer I Anti-EM Screen Reinforcer I Dropped items: Bloodclaw Light Missile, Qty: 216 'Arbalest' Assault Missile Launcher, Qty: 5 Large F-S9 Regolith Shield Induction Ballistic Control System II, Qty: 2
Websites will take these killmails and break them down, assigning points relating to the value of the ship destroyed, and the contribution of each assailant. Depending on the complexity of the killboard, these points may relate to an in-game rank, allowing more skilled players access to higher level alliance functions. So as you can understand, many players were suitably upset when they faced the potential of loosing the last 3 months worth of work.
Going further into the tangent, the developers of Eve Online have been increasingly generous to 3rd-party developers. They have always made their game data available to developers who want to use it (within reason), and have recently made a REST API available, allowing developers to directly access in-game statistics.
Using this API, in addition to the data feeds made available with most popular killboards, the alliance was able to recover its most recent killmails. However, like most API’s and data feeds, this data was incomplete – only covering the last 14 days. A way to crawl further back was required.
Where I cone into this, and the crawling process itself, is covered in part 2.
Final note. The title of this post was inspired by the most excellent Bill and Ted.
