IIS4/IIS5 were aimed at classic ASP, which used CDONTS to send emails. CDONTS wasn’t aware of external mail servers, it assumed you’d run a single-box environment, so is hardcoded to use localhost and the cutdown SMTP service bundled with IIS: http://classicasp.aspfaq.com/email/can-i-use-a-remote-smtp-server-with-cdonts-newmail.html

IIS6 and above use the replacement, CDOSYS, which is aware of remote servers, so you don’t necessarily need your web server to run a local SMTP service: http://www.asp101.com/articles/john/cdosmtprelay/default.asp

IIS can still run a local/cutdown SMTP server, and this can still be managed via IIS.

IIS7 seems to further this concept and do away with local/cutdown SMTP service under IIS altogether. Although SMTP server can be installed on Windows2008, it’s considered a separate product (and not manageable from IIS7, IIS6 is used with it’s compatibilty mode); Instead, IIS has the option to use an external SMTP server.

http://weblogs.asp.net/steveschofield/archive/2006/12/19/iis7-post-23-vista-and-smtp-server-where-is-it.aspx (Note: Vista runs same ‘core’ as Windows 2008)

Just to give a recap on what I’ve been upto:

• I am now 29.
• I now know a little bit more about SQL Server and Windows Server, and a fair bit more about IIS6.
• I went to Bovington Tank Museum
• I had a drive about in a  Rally Car at Silverstone.
• I saw Blur play in Hyde Park

Anyway, the aforementioned teahouse can be found in this view. You’ll want to stand at this point, to catch geisha entering/exiting the teahouse, geisha coming from the alleyway, and geisha coming from behind.

Check it out.

Geiko (Geisha)

Originally uploaded by mindfieldz

I was recently looking over my most popular Flickr photographs, and I came accross this one, shown to the right.

I visited Japan in May 2008, and one of the biggest highlights was seeing a real geisha in Kyoto (not to be confused with normal Japanese women in nice kimono’s). Lonely Planet, and Rough Guide don’t really give you an indication of where to find geisha, only that they’re “in Gion”.

After some frantic goggling, in a ryokan in Kyoto, I eventually found hints tucked away in a photography blog. Matching against a Japanese google maps, I convinced my wife to follow me to a side street in Gion.

Success! I’d managed to find the Ichiriki-tei Teahouse, the most prestigious teahouse in Gion.

[flickr style="border: 1px solid #5A5A5A"]set:72157615254849529[/flickr]

So, if you want to get awesome photos of geisha, follow the instructions below.

1. Get yourself to Gion, in Kyoto.
2. Locate the Ichiriki-tei Teahouse, this can be found just off the main street in Gion. On google maps, it’s the C-shaped grey building.
3. Wait until about 5:30pm-7:30pm local time. Don’t get too close to the entrance of the teahouse, else someone will come out and ask you to move away from the entrance. Taxi’s and other vehicles pull up to the etrance, so be careful if you are attached to a camera lens.
4. You will know if you are in the right place at the right time, as a fairly large crowd will gather.
5. Best place to position yourself is at the corner of the teahouse, looking down the alleyway behind the building. Geisha will come from all directions!

That’s it. In about an hour I saw 7 geisha, which is not bad considering it was a random tip-off from another blog (whose address I’ve long forgotten).

[Edit: Just realised that latest WordPress upgrade managed to remove the photos I'd posted.]

In terms of the kit used to get the shots above:

Canon EOS 400D
18-55mm f3.6-5.0 Tamron lens.
55-200mm f3.6-5.0 Tamron lens.
Mostly 1/60 to 1/200 shots, handheld.

Be prepared to do a lot of running with a camera.

There’s a good reason, seriously…

Anyway, I have made some tweaks to the site.

• The site now defaults to 75% width, up from 70%. I had aimed/hoped for 80-85%, but my header images aren’t wide enough, expect that to change at some point.
• My code boxes now word wrap, so you’ll be able to see my code in all it’s glory.

Not much, but a start. And some content.

Although SparkFun says that they’re out of stock, it seems to be more of an ‘order on demand’ type of thing. I purchased a Rootooth on backorder, and had one posted to me within two weeks. For reference I chose the cheapskate AirMail option with no tracking, and the item turned up within about 3 weeks from my original order date. Although SparkFun had marked the box belot its actual cost, Customs had decided to open and hold it for a few days anyway.

The Rootooth itself. Don’t be disheartened if it doesn’t work out of the box. I discovered three things with mine:

1. You need a Bluetooth dongle that works on Widcomm or the Windows Bluetooth Stack (preferred). I initially tried it with a cheap Bluesoleil Bluetooth dongle, and it failed miserably – Vista refused to connect to the virtual serial port on the Rootooth, and XP would consistently drop the connection every few minutes.
   I eventually bought a Belkin Bluetooth dongle for £10, the F8T013UK. Although it has Widcomm drivers, you can get it running under the Windows stack by following some simple instructions.
2. The Rootooth has been upgraded to use a Firefly Bluetooth module. This version is rumoured to set an incorrect baud rate for the 5xx series of Roombas, despite using the SU,115K instruction. To fix, send the following commands to your Rootooth:
   $$$
   U,115K,N
   Your Rootooth should return back to normal mode (not command mode), and you’re ready to go.
3. There are few websites/forum posts giving information relating to the Roomba 5xx series, even fewer bits of code/software which work with them, and even fewer that work with both.
   I discovered that the 5xx series uses a different set of codes, and that one person had coded for the latest Rootooth/Roomba combination.

So, my Roomba now goes backwards and forwards, and I can monitor the sensors. Next stop, Roomba/Twitter and Roomba/Wiimote.

Although the collection process outlined in part 2 is still running, it is estimated that we’ll end up with approximately 70,000 raw killmails which mention our alliance (from the predicted total of 6million). Looking at the killmails themselves, many are duplicated. This is because both the victim, and the pilot who landed the final blow, receive a copy of the killmail; and where more than one alliance is attacking, each alliance will copy the killmail to their own killboard. Of the 70,000 killmails, approximately 25,000 are duplicates, and can be ignored.

De-duping SQL tables is a bit of a nightmare prior to MSSQL 2005. You have to produce a subquery in order to find the latest record for each group, and then delete all others. It’s fairly prone to error if you’re not sure what you’re doing. Fortunately, MSSQL 2005 solves this problem with the introduction of the RANK() keyword. RANK() will give an integer column of where a particular record ranks within a group, for example:

SELECT kID, iID, killmail, posted,uploaded, RANK() OVER (PARTITION BY SUBSTRING(killmail, 0, 50) ORDER BY kID, iID) AS rnk FROM ekKillmail

A sixth column, rnk, will rank the records in order of kID and iID. We only group on the first 50 characters of the killmail, as this speeds things up slightly. The timestamp, victim, and lost ship are covered in these first 50 characters, and provide a unique enough record identifier.
In order to get the first record for each group, we push the above query into a subselect:

SELECT kID, iID, killmail, uploaded FROM
 (SELECT kID, iID, killmail, posted,uploaded, RANK() OVER (PARTITION BY SUBSTRING(killmail, 0, 50) ORDER BY kID, iID) AS rnk FROM ekKillmail) AS k
 WHERE k.rnk > 1

This instantly de-duplicates our killmails, giving us about 50,000 killmails to process. I should mention that there’s an excellent guide to the RANK() keywork on Rafeal Salas’ blog.

We can reduce this 50,000 figure further still by filtering records on date. As mentioned at the start of this story, only 3 months of records were lost, whereas we’re requesting killmails from other alliances’ killboards for as long as they’ve been active. Looking at the data we’ve gathered, some killmails are as early as 2005. Although each killmail begins with a timestamp of when the kill happened, this is stored as text, and is very difficult to filter on. This is a relatively easy thing to fix, as we can simply cast the first 17 characters (which represent the date/time), into a datetime data type.

UPDATE ekKillmail SET posted = CAST(SUBSTRING(killmail, 0, 17) AS DATETIME) WHERE posted IS NULL

Meaning we can expand our killmail query to then filter on the date and time of the kill (posted column):

SELECT kID, iID, killmail, uploaded FROM
 (SELECT kID, iID, killmail, posted,uploaded, RANK() OVER (PARTITION BY SUBSTRING(killmail, 0, 50) ORDER BY kID, iID) AS rnk FROM ekKillmail) AS k
 WHERE k.rnk > 1
 AND k.posted > '2008-09-24 00:00'
 AND k.posted < '2009-01-06 00:00'

This reduces the number of killmails we have to upload back to our alliance killboard to a much more manageable 5,000. We now need to tackle the uploading process.

Uploading killmails manually to our alliance’s killboard is a fairly rare occurrence, the website itself uses Eve Online’s API to make frequent queries on each pilot’s kills, as well as pull kills from other alliances’ killboards. However as we learnt in part 2, these data feeds only go back a few days, or shorter. Prior to the Eve Online API, pilots had to copy and paste killmails from their Eve Online client, into a web form on the alliance’s website. This form is still available, however it is restricted to authenticated users – preventing random internet users from posting fictitious killmails.

Authentication is based upon VBulletin, which uses cookies and sessions to keep track of the user. Firefox’s Web Developer add-on will allow you to inspect cookies within a given session, so its then just a case of copying them into our C# page request/post.

        static void writeKills()
        {
            //Declarations
            SqlConnection DBConn;
            SqlDataAdapter DBAdapt;
            DataTable DBTable;
            int i;

            //Connect to DB
            DBConn = new SqlConnection("Data Source=SERVER\\SQLEXPRESS;Initial Catalog=eveKills;User Id=sa;Password=secret;MultipleActiveResultSets=True;");
            DBConn.Open();

            //Get killmails
            DBTable = new DataTable();
            DBAdapt = new SqlDataAdapter("SELECT kID, iID, killmail, uploaded FROM (SELECT kID, iID, killmail, posted,uploaded, RANK() OVER (PARTITION BY SUBSTRING(killmail, 0, 50) ORDER BY kID, iID) AS rnk FROM ekKillmail) AS k WHERE k.rnk = 1 AND k.posted > '2008-09-24 00:00' AND k.posted < '2009-01-06 00:00' AND uploaded IS NULL", DBConn);
            DBAdapt.Fill(DBTable);

            //Loop through killmails
            for (i = 0; i < DBTable.Rows.Count; i++)
            {
                //Process killmail
                postKill(DBTable.Rows[i]);

            }
            //Close DB Connection
            DBConn.Close();
        }

        static void postKill(DataRow drKill)
        {
            //Declarations
            HttpWebRequest HRequest;
            HttpWebResponse HResponse;
            StreamReader RResponse;
            Stream Request;
            SqlConnection DBConn = null;
            SqlCommand DBComm;
            string sResp = null;
            string sRequ = null;

            //Create HTTP Request
            HRequest = (HttpWebRequest)WebRequest.Create("http://www.iron-alliance.com/kb/post.php");
            //Emulate an authenticated session
            HRequest.CookieContainer = new CookieContainer();
            HRequest.CookieContainer.Add(new Cookie("bbuserid", "secret", "/", ".iron-alliance.com"));
            HRequest.CookieContainer.Add(new Cookie("bbsessionhash", "secret", "/", ".iron-alliance.com"));
            HRequest.CookieContainer.Add(new Cookie("bbpassword", "secret", "/", ".iron-alliance.com"));
            HRequest.CookieContainer.Add(new Cookie("bblastvisit", "secret", "/", ".iron-alliance.com"));
            HRequest.CookieContainer.Add(new Cookie("bblastactivity", "0", "/", ".iron-alliance.com"));
            HRequest.CookieContainer.Add(new Cookie("vbulletin", "secret", "/", "www.iron-alliance.com"));
            //This is a POST, so change header accordingly
            HRequest.Method = "POST";
            HRequest.ContentType = "application/x-www-form-urlencoded";
            //POST Request Payload
            sRequ = "killmail=" + drKill[2].ToString() + "&Post=Post";
            HRequest.ContentLength = sRequ.Length;
            //Make POST            
            Request = HRequest.GetRequestStream();
            Request.Write(Encoding.ASCII.GetBytes(sRequ), 0, sRequ.Length);
            Request.Close();

            //Read Response
            HResponse = (HttpWebResponse)HRequest.GetResponse();
            RResponse = new StreamReader(HResponse.GetResponseStream());
            sResp = RResponse.ReadToEnd();

            //Check response for keywords
            if (sResp.IndexOf("Success!") > -1)
            {
                //Kill posted successfully
                sResp = "Success!";
            }

            else if (sResp.IndexOf("already posted") > -1)
            {
                //Already posted
                sResp = "Duplicate!";
            }
            else if (sResp.IndexOf("problem!") > -1)
            {
                //Problem
                sResp = "Problem! " + sResp.Substring(19000, 1024);
                //Write the killmail to a text file, so that we can deal with it later
                File.WriteAllText(drKill[0].ToString() + "_" + drKill[1].ToString() + ".txt", drKill[2].ToString());
            }
            else
            {
                //Other (not successful)
                //Write the killmail to a text file, so that we can deal with it later
                File.WriteAllText(drKill[0].ToString() + "_" + drKill[1].ToString() + ".txt", drKill[2].ToString());
            }

            //Connect to DB
            DBConn = new SqlConnection("Data Source=SERVER\\SQLEXPRESS;Initial Catalog=eveKills;User Id=sa;Password=secret;MultipleActiveResultSets=True;");
            DBConn.Open();
            //Update killmail
            DBComm = new SqlCommand("UPDATE ekKillmail SET uploaded = GETDATE(), response = @P1 WHERE kID = @P2 AND iID = @P3", DBConn);
            DBComm.Parameters.AddWithValue("@P1", sResp);
            DBComm.Parameters.AddWithValue("@P2", drKill[0]);
            DBComm.Parameters.AddWithValue("@P3", drKill[1]);
            DBComm.ExecuteNonQuery();
            //Close DB
            DBConn.Close();
        }

Finally, we read back the response of the killmail posting. The Eve Dev killboard gives fairly good feedback on how a killmail posting went, so we can search for the phrase “Successful”,  “already exists”, or “problem” to determine the status of the posting. These messages correspond to a successful posting, a duplicate killmail being found, and a failed posting. In our code, we’ve caught the failed postings, and written the killmail out to a simple text file, so that a human can determine the cause of the problem. We’ve also written the status back to the database so that we don’t attempt to resubmit problematic killmails.

That’s it! Overall, about 5,000 killmails were recovered, with about 400 failing to submit to the alliance killboard. It took approximately 72 hours to mine all the original killmails, partly due to throttling on the remote web servers (at its peak, I was downloading killmails at a rate of 300kb/s), and approximately 2 hours to filter and submit killmails to our killboard.