We all know what’s wrong with Microsoft certifications. The multiple-choice format means that people can cheat too easily, and over the years, the questions have often felt too specific, asking the kinds of questions that proper IT professionals just look up in Help systems like SQL Books Online.
To help address this problem, Microsoft started to come up with simulation questions. They were used in some Windows NT exams, and most notably for me (as I helped write them), in the core SQL Server exam 70-431. These were Flash-style applications designed to look and feel like the real applications. This is good, but they’re not error-prone (people who have sat 70-431 will know of a particular error in one of the drop-down boxes), and they can only really test usage of the UI. Definitely an improvement on multiple-choice though, and when writing these questions, special effort was made to find things that would make cheating very difficult.
The biggest problem with simulations is that people have different ways to achieve their goals. “More than one way to skin a cat” as the expression goes. This is increasingly so with technologies like PowerShell coming into almost every area of server administration, and particularly applies to developer exams where the goal should be achieving something to satisfy a unit test rather than answering a particular knowledge question.
And especially for SQL Server. Most DBAs will use T-SQL to perform the tasks they do on a daily basis. Some will use Management Studio, others will use sqlcmd, or pre-created scripts. Recently, quite a few people will have started using PowerShell, particularly if they are already using PowerShell scripts to maintain Exchange and Windows. Therefore, testing becomes more difficult.
70-113 fixes this problem completely. Whilst I don’t expect to have passed (it’s an exam about Active Directory, which I only know a little about), I thoroughly enjoyed the overall experience. It asked me to configure a couple of servers according to a set of instructions, and then actually gave me connections to the machines. And they were complete machines. Obviously I didn’t have Internet access, but I did have the Windows Help system. This alone would have got me past a few hurdles, as I could look up a few things that I couldn’t quite remember.
With SQL Server, examinees will have SQL Server Books Online available, but that’s like it is in the real world. If asked to create a particular type of trigger, you can remind yourself of the syntax for that. If asked to make sure that a backup uses the COPY_ONLY option, then you can look up where that goes. But this is the problem. In 70-113, the information provided seemed to give away a little too much. It explicitly told me what to use for many of the options, but I would’ve preferred to have had it describe something akin to “Make sure that the full backup you take doesn’t affect the next day’s regular differential backup” rather than “Use the COPY_ONLY option”. This way, it can test the knowledge of the system, rather than whether or not you can find the appropriate checkbox.
The other area that I would like to see is a combination of question-answer and virtual lab. I’d like to be given the connection to the server, have to configure various things, but then also answer questions. “How much free space is there in this file?”, “How many times has the index with IndexID = 3 on table X been scanned?”, etc. This would not only test whether you know how to configure the system, but also whether you know how to find information – a very important skill which isn’t really tested yet.
70-113 is definitely a step in the right direction, and I encourage everyone to give it a try (today is the last day you can register for it). Don’t feel like you need to pass, just do the exam and provide comments about what you think.
This Post Has 5 Comments
Good post, Rob. I’m excited about this new format, as it shows that you have to know how to actually do the tasks, and under time constraints.
Any idea on the marking system? Is it graded immediately (which I would assume to be marked as just checking to make sure the VMs are left in the correct state), or is there some amount of marking performed by people? The reason being, in a practical exam, you might get 90% of the way there, but get 0 marks for the task. For example, you had to take a full backup and put it in a directory.
What if you took the backup correctly, but named the file slightly differently from what you were asked? Is that worth a 0? At least the multiple choice questions are an obvious right/wrong.
Sorry Jim, I’m not sure on that one. I can almost certainly say there won’t be marking by people, but I don’t know what kinds of things it will be based on.
If you’re asked to do a backup, it could check the date of the last backup to see if you achieved that correctly. Hopefully you can be involved in creating questions as well, so that you can make sure that the checks cater for those scenarios.
Rob
@Jim: “What if you took the backup correctly, but named the file slightly differently from what you were asked?”
I’d expect you’d fail that question.
Not being able to adequately follow instruction, or insufficient attention to detail, or thinking they know better (pick one, or more) are more fundamental than anything specific to an operating system let alone a backup. If I were interviewing based on a scenario, inability to follow specific arbitrary instruction is a most telling fault, the explanation why often more telling again 😉
I think it will all come down to whether the question has multiple parts to it (which I’m sure it will). If one part is “Make a backup”, and another is to put the file somewhere in a particular drive, it may not care whether you have the backup go straight there or whether you take the backup and rename the file.
But I can’t be sure, and if I’m involved in creating questions, I won’t be able to say anyway.
Rob
right on target with this one. Too easy to cheat. They need to take a page from the cisco CCNP testing. Part of it is multi choice. Other part is a lab. You have to setup the equipment according to specs. Then they break what you setup and you have to come back in and fix it.