(Still on the topic of having community-written exam questions, which results in a larger pool questions to use on each topic)
Every so often, I get asked about whether exams are adaptive. So, you get asked a question, and the next question is harder if you got it right, or easier if you got it wrong. Or maybe it asks you five questions at a particular level, and then five questions at the next level, until it works out what level you match best. Then you either pass or fail depending on what level you finish at. And by ‘finish’, this is just having the system determine “Yes, you’re better than the passing grade”, or “Sorry – you’re not quite there.” You might only be in there for 10 minutes, but if you did well, then you’re given a pass as soon as you demonstrate that you really know your stuff. I guess this is explained better by Wikipedia.
The biggest problem with this approach is that you need a lot of questions. Suppose you have 6 ranks of question. You want quite a lot of questions. Especially since different people would rate questions in different ways. In 70-445, which covers SSIS, SSAS, SSRS and Data Mining, an expert in SSIS will find those questions easier. Having a larger pool of questions means that on average, you’re going to be able to grade questions easier. If you put some ‘easy’ questions into a harder rank, that’ll be compensated by the harder questions that put into the easy rank.
I like the idea of adaptive questioning. But I think it’s really tough to do, and still works better when you’re being assessed by a person, not a computer. If they could have way more exams (but smaller), with way more questions in each one, then they could have a 30-minute test to see if you knew your stuff. Even if it wasn’t adaptive, this idea would work. But if you’re trying to introduce something like adaptive, you need smaller areas of study. You can’t just have a tiny exam on “Web Development”, but you could possibly have one on XAML. You couldn’t have “Implementing and Maintaining SQL Server”, but you could have one on “High Availability in SQL Server”.
Just food for thought of course… I discussed this type of thing with Howard earlier in the year too.
