I was home from work slightly later than usual today. Normally this is down to managers bursting in at the last minute demanding that I look into something, sometimes it’s due to me getting carried away in the moment and not realising the time. Today it was because there was an online assessment snafu.

It’s an event I knew about, it’s an event I’d planned for, it’s an event that I’d added to my calendar. It’s an event I completely forgot about because:

1. I added it to my, and my colleagues’, calendars a day late. Call this a moment of insanity if you will.
2. I forgot to set a reminder.

Anyway, at quarter-past-four, a rather anxious member of a department’s admin staff comes into the office, asking where I’ve been for the past fifteen minutes. Cue a hurried rush downstairs, to see the academic who fired off the original enquiry coming to the end of her introductory slides, and a room full of student’s at computers.

A silence descends on the room. Everyone in the room turns to look at me. No idea why. I’m here purely to see what’s going on. Perhaps I should explain.

We use a fairly well known online assessment package. We’ve been using it for nearly eight years now, and I’m fairly adept with the in’s and out’s of it, having spent my first 12 months of employment developing against it’s API. One of the things we use it for is online examinations, it’s excellent for large multiple choice papers, and its more eco-friendly than the old Optical Mark Reader (OMR) that is still in widespread use. The main bonus of this product over it’s competitors, is that it can lockdown the computer during the assessment; no Wikipedia in the background, no MSN, no email. Although the software has its problems, it’s actually rather good on the whole, and I just managed to talk the establishment round to going site-wide with it’s installation (this will come back to haunt me, you heard it here first).

So, the department in question, boosted by our general murmurings that this was a ‘supported system’, decided to approach a 3rd party company which resold the same online assessment system. There is a method to this madness, the company owned the intellectual property rights to a fairly clever assessment and its questions; so unless we forked out several thousand pounds for use of the IPR, it was vastly cheaper to use their service direct. So, department buys rights to use this software on 100 or so students, and a week before it’s due to happen, the staff member responsible for coordinating it all goes off on long-term sick.

Enter our team, stage left, pursued by bear.

The end result being that I left work about 25 minutes later than usual, and spent just over an hour helping a traumatised department with it’s first forays into the world of online assessment. Overall we did good, although there were one or two moments where my colleague and I exchanged thousand-yard-stares across the melee.

Hopefully the department will send a thankful email tomorrow, they’ll realise the benefits of online assessment, and we’ll have another one on the e-learning bandwagon. On our side of the fence, it was a reassurance that the quality assurance steps we had in place weren’t a simple folly, and actually do help in these situations.Which brings me to the meat in this sandwich. Top-tips for online assessments. In no particular order:

1. Test, test, and test again.
   Academics make mistakes, we (as admins of the system) make mistakes, computers (without question) always make mistakes , everyone makes mistakes.
   1. Give the academics ample time to test, and explain the consequences if they get it wrong.
      In our case, once a mistake has gone ‘live’, we say that it cannot be corrected (for QA reasons), and that the academic has to make any marking corrections themselves. Only make changes on the day of the exam if you are sure its the right thing to do, or there has been a monumental cock-up; always seek a colleague’s advice on this (preferably a sane one).
   2. Ask a colleague when you want to cowboy something, or something simply doesn’t appear to be ‘right’.
      Maybe it’s because it’s a Monday, maybe it’s down to the lack of caffine, maybe it’s the academic whose made a mistake. However it’s always good to get a sane voice on any matter which needs thinking about.
   3. Test the system, test it again, go back and test it a third time to catch it by surprise.
      Computers are evil. We’ve had a situation where a completely unrelated routine update installed over the weekend, has wrecked the way that online assessments handle mathematical equations. We now spot-check six computers in the week leading upto the online exams, and ask the rest of our section to politely not touch anything.
2. Simple is good.
   Despite the option to ‘make it better’, don’t. Do the simplest thing possible and stick to it. See pretty much every other item on this list for an explanation why.
3. Don’t underestimate the onion of stupidity.
   Despite whatever attempts you make to ensure the system is foolproof, people will find imaginative ways to break it all. The best example I can give is a student who managed to “kick the plug out of the wall socket”, thus powering down his computer. Words fail me.
4. Give them a dry run.
   If students have not used the system before, give them the opportunity to try it out. Even if it’s a simple assessment containing questions like “What colour is the sky?”.
   We normally couple this with point 5 and 7 below, and it helps overcome some of the consequences of point 3 above.
5. Write it down for them. With pictures.
   There’s a lot of reasons why students may not follow the instructions you give them, many are valid but some fall under the umbrella of the ‘onion of stupidity’ mentioned above. Writing the instructions down helps everyone:
   1. It helps stagger the start of your assessment, meaning that individual problems don’t all happen at once, and the system gets a bit of a rest. Explain to students that they all have individual timers, so nobody is disadvantaged.
   2. English isn’t everybodies first language, and some students struggle with reading. Simple instructions and diagrams help. Get them down to a single sheet of A4, picture heavy.
   3. If it all goes wrong, you can point to a piece of paper, and walk away.
6. Let everyone know what their role is.
   Myself and my colleagues are not invigilators, we don’t want to be. Equally, most invigilators aren’t technically trained, and don’t really want to explain to a student why their user account has expired (see point 7 below). I’ve even been kicked out of an online examination I was assisting with, as an invigilator mistook me for a student.
   Make sure everyone knows their role prior to the start of the online examination.
7. Make sure you hammer home the rules, and don’t be soft over the exceptions – they have been warned.
   Rules exist for a reason, make sure you have rules and you follow them, else people will loose faith in both you and the system.
   1. Staff need to know the basics of the system. What it does, how it works, where it’s most likely to fail, where students are most likely to have problems (or create them). You’ll find that non-technical staff are happy to deal with the little things if they know about them in advance, and this is invaluable during the first 15 mintues of an online examination.
   2. Students need to know what’s expected of them, and the consequences if they get it wrong.
      Not only are students told that this is like any other examination (eg. mobile phone = evicted from exam), but we tell them three-fold that online examinations are slightly different to the norm. Students are informed verbally by their lecturer, they are informed by email, and they are informed during their dry run (point 4 above). We tell them:
      • It’s the student’s responsibility to be familiar with the system. We’ve given them a dry run which is as close to the real thing as possible, and they have unlimited attempts.
      • It’s the student’s responsibility to make sure their computing account is in order. No changing password 10 minutes before the exam, no turning up and claiming that you’ve ‘forgotten’ your password. No account, no exam.
      • This is an exam, no really it is. Although myself and my colleagues aren’t invigilators, we’ll still give you a hard stare for talking, and we’ll still point you out to someone who has the authority to do something about you.
      • Follow the instructions provided (point 5), to the letter. You should log in, visit the webpage provided, take the online examination, and log out. Visiting Facebook isn’t mentioned on those instructions, we’ve checked. Twice.
8. Always have a ‘Plan B’, even if ‘Plan B’ is ‘run away’.
   There are always going to be situations beyond your control. Power cuts do happen, people do set of fire alarms, workmen to drill through gas pipes in the adjacent room (ok, I made that last one up). Despite all the planning and preparation, you should always have a contingency slot. You’re not going to make any friends if you’re forced to use it, but it’s the elephant in the room should anything go wrong. Saying that, we’ve gone four years now without need of this contingency slot.
9. It will be hectic in the first 15 minutes.
   Ok, not a rule exactly, but a piece of advice. The first 15 minutes will go something like your worst ‘disaster recovery’ scenarios. You’ll get used to it. Honest.

So, yeah. That’s about 1,500 words on why online examinations need a level of quality assurance. It’s also my first work related post on my blog. So once I’m comfortable with this new format, expect some more offerings. I’m off now, as Cat #3 is mewing desperately and looking for a cuddle.