Is the Institute of Masters of Wine fit for purpose?

It’s hard to imagine Jancis Robinson doing a stand-up routine at London’s Comedy Store isn’t it?  Yet, after reading her take on the recent Master of Wine (“MW”) examination papers it would seem to me that she has all the hallmarks of a good humorist. 

There was one routine in particular that I found amusing where she published the latest MW exam papers on her website, and then commented on the questions each MW student had to grapple with.

While referring to the MW exam questions, Robinson wrote:

“I should imagine many wine lovers will read them and think, 'I could answer those'. But I can assure you that the questions always look deceptively straightforward. The examiners are looking for extremely detailed responses that the questions don't really hint at.”

Amusing indeed, at least it would be if she was joking.

Let’s just read between the lines and understand what Robinson is really saying.  The first point: “I should imagine many wine lovers will read them and think, 'I could answer those'”. 

From that we can assume the questions were fairly straightforward and, indeed, Robinson admits many people could answer the questions.  Great!  So loads of people should pass the MW exam then, right? Wrong!

She goes on to say: “But I can assure you that the questions always look deceptively straightforward”.

So what are we to understand from that?  Only one thing, really -- the questions are designed to deceive.  Why is that?  Why should an examiner wish to mislead a candidate with a question?  I thought the whole purpose of a question was to be clear in its intention.  The definition of Question is “a sentence worded or expressed so as to elicit relevant information”.  It says nothing about being deceptive, so why would the MW examiner wish to be deceptive? 

But wait, Robinson goes on to explain why the question is purposely misleading: “The examiners are looking for extremely detailed responses that the questions don't really hint at”. 

Put another way, Robinson is saying the examiner is looking for a detailed answer to a question that has little or nothing to do with the actual question they are asking! 

It would seem that not only do students need to know every finite detail about every aspect of wine but, to qualify as a Master of Wine, candidates are required to be mind readers too.  Well that explains everything doesn’t it?  The Institute of Masters of Wine (“IMW”) makes Harry Potter’s time at Hogwarts look like a walk in the park.

This old-school approach to an exam, regardless of the subject, demonstrates profoundly that the IMW is an organisation trapped in the dark ages.  This out of date attitude reinforces the point of view that the MW exam is totally irrelevant to a modern day wine industry.

Ask any major educational exam body in the world today, and they will tell you that the accepted standard for setting exam questions requires that exams adhere to a number of fundamental principles.  Two of the basic principles include:

1) Clarity of the questions; and

2) Fairness in the examiners’ approach. 

The MW exam clearly fails on both of these principles.

The IMW fails by Robinson’s own admission on clarity because, as she so eloquently put it, the exam questions are designed to deceive.  Moreover, by virtue that “the examiners are looking for extremely detailed responses that the questions don't really hint at” the exam fails the fairness test too. 

Examiners must give candidates a fair chance to demonstrate their knowledge and give students the chance to succeed.  If examiners ask one question, but really want the answer to another question they’re not even asking, how can that be objective?  How can that be fair and just?  It is certainly not being clear!

The IMW website contains several videos with post-mortems on past questions indicating what the examiner is actually looking for, and it is interesting to see the length of explanation the examiner goes to in order to explain the question.  What is most revealing is that the explanations in many cases bear little resemblance to the brief question posed in the exam.  This indeed confirms Robinson’s assertion that the examiner is giving little away in terms of what response he or she is seeking to solicit in a textbook answer. 

The examiner says several times in the videos “It’s not rocket science”. Dammed right it isn’t!  Yet the IMW appear to go to great lengths turning a fairly straight-forward subject of wine into rocket science. Indeed, there are fewer MWs in the world today than there are qualified rocket scientists.

The exams appear less concerned with the knowledge students have, but rather more focused on the students’ ability in decrypting and second guessing the vague questions posed. 

The crunch of the matter is, the examiners are looking for very specific detailed answers to extremely vague questions.  That is not a true test of a student’s knowledge, and that is not the real purpose of any exam worthy of respect -- nor should it be. 

Given Robinson’s mind-set, which presumably reflects that of the Institute in general, coupled with the indisputable evidence that only 366 people worldwide have ever passed the exam in its sixty year history, what more evidence does one need in order to conclude that the Institute of Masters of Wine is an old fashioned, irrelevant organisation, not fit for purpose?

While the IMW openly publishes the number of candidates that have passed the exam, they appear less open about pass rates and indeed the number of candidates that have failed the exam or dropped out prior to completing the course.  I suspect this number is substantial and would likely prove embarrassing for the organisation.

The IMW is little more than an elitist club, accessible by invitation only, designed to keep the riff-raff and rabble out.  But other than being a club where members can go off on various jollies comparing each other’s tasting notes, I see no useful or meaningful purpose to the organisation whatsoever.  Indeed, the institute is not relevant to a modern day wine industry. It has certainly failed in its original remit to “improve the standard of education within the British wine trade”.

The proof of this is in the job market.  There are almost no jobs where the MW qualification is a requisite, hardly surprising given the insignificant number of candidates that have ever qualified in its history.  So what purpose does the exam have?

While there is no question the IMW has an abundance of talented wine specialists, what it appears to lack are qualified educationalists.  This state of affairs may go some way to explaining why the institute has created a pretty meaningless examination system.  While the IMW website boasts an external education advisor, I cannot find evidence that the people who create or mark the exams are true educationalists. 

When creating an educational system, such as the IMW, it is critical to have qualified expertise on modern day educational systems, theories and concepts.   Indeed, these skills are arguably as important, if not more so, than expertise on the subject matter in question.  But ideally, any educational body should have a good balance of all the appropriate skill-sets required. 

The IMW ship appears to be overloaded with wine theorists, but limited or no real educationalists to speak of.  If the captain in charge of this small vessel cannot counter balance the crew with adequate educationalists, she may as well scuttle the ship altogether for all the good it can do in its current form.

Readers' comments (22)

  • This article is based on the author's interpretation of another person's statement and a few videos. This is just not enough to make a serious statement about the imw. Especially if the author makes it sound like he is bitter about being rejected by the institute...

    Unsuitable or offensive? Report this comment

  • Dear Mr Konstantin Baum,
    Thank you for reading my blog and for taking the time to provide your feedback. Your feedback is very much appreciated and it is great to spark a healthy debate on this intriguing subject.

    Just to set the record straight, I am not as you say “bitter at being rejected by the institute”. The institute has never been offered the opportunity to reject me, as I have never applied to enter the programme nor have I ever contemplated the idea at any time.

    The article is based on my 50 years of life experience, including a 10 year period at the helm of a wine retailer and importer that I founded.

    During the course of running my business I have met numerous fellow wine industry colleagues over the years, and I have naturally encountered many people who have been involved with the IMW in some form or another. Over this 10 year period, having listened to countless conversations as well as performing my own research, I have formulated real concerns as to the fairness of the MW exam. It is primarily from this experience that I make my arguments and draw my conclusions.

    The focus of this particular article was indeed on Jancis Robinson’s commentary of the current exam, as her comments provided the real evidence that substantiate my long held theories and views. However, the research and arguments that delivered this conclusion were carefully considered and formulated over the last decade.

    The statement by Jancis Robinson is not just “another person’s statement” as you argue in your comment. Robinson is as authoritative a spokesperson of the IMW as any you’ll likely find. There is no doubt her article reflects the views of the Institute.

    The indisputable fact is only 366 people have passed the exam over a sixty year time period -- less than a 10% pass rate. Given the extremely high entry requirements to enter in the first place, this is a shameful record and would it suggest one of two things or a combination of both: either the teaching is at fault, or the examination is at fault. For the reasons I've outlined in my article it would appear to be the examination.

    What you fail to do in your feedback, with the utmost respect, is to provide counter arguments to the valid points I raise. Thanks again for your feedback.

    Unsuitable or offensive? Report this comment

  • I was not intending to provide counter arguments in my previous comment - I only wanted to say that I think that this article is poorly researched. The tone of the piece and the lack of factual evidence make it sound like you are holding a personal grudge (not to say you do).

    First of all: I did not say that JR is "just" another person - I said that she is another person and you are interpreting her comment. She did not say that the questions are designed to deceive - she says that they look deceptively straightforward when addressing her readers (or "wine lovers").

    The exam questions might deceive the unprepared reader of JR.com but as a student you actually go through a programme that is designed to prepare you for sitting the exam. I doubt that many people enter the examination room thinking they can pass the exam just because they love wine...

    In the programme you learn how to approach a topic from different angles, how to provide factual evidence and examples for the points you are making and not to jump to conclusions but allow the facts to speak for themselves. All valuable lessons, especially for aspiring writers...

    All of your following assumptions (that the IMW is "trapped in the dark ages", "that the MW exam is totally irrelevant to a modern day wine industry", that the IMW is "an old fashioned, irrelevant organisation, not fit for purpose") are based on your questionable interpretation of one of the most prominent MW's comment, which you call "the real evidence".

    You might have a lot of experience in the trade but you fail to use the level of factual evidence that is required when writing a quality piece for a quality publication such as Harpers. An interview with JR to get confirmation on your interpretation of her statement would be the minimum in my view.

    The Master of Wine is difficult and that is why there are so few MW. The programme is not perfect and the IMW needs to constantly work on improving it, as they do. The qualification is still very relevant in the industry (why would you otherwise write about it?) as many of them are opinion leaders in their markets and hold important positions in the wine trade.

    You should be aware that several of the most influential writers (some of which work for Harpers) are MWs or are studying to become one. I am sure that your colleagues would prove to be a valuable source of real insight for your next article on this topic.

    Sincerely, KB

    Unsuitable or offensive? Report this comment

  • Thank you for your feedback Mr Baum. You make some good arguments and I’d like to address these.

    The tone of the piece is calculated to be provocative. This style of writing is designed to spark debate. The fact that you have kindly taken time to respond would suggest that it has had the desired effect.

    In some respect I am playing devil's advocate. In all fairness it is no real concern to me that someone may waste huge sums of money, and a large part of their life, studying for an exam where 90% have absolutely no chance at passing – ultimately that is their choice.

    As someone completely detached from the IMW with absolutely no aspirations to become one, arguably that makes me the ideal independent commentator on the subject. I certainly have no axe to grind.

    The article is in fact a blog and an opinion piece and is therefore not subject to the same rigorous scrutiny that I am sure you have been trained to when writing a dissertation for your MW exam. I am expressing a point of view for the purpose of debate. But I do take on-board your request for further evidence in future articles.

    If, as you say, the MW “programme is designed to prepare you for sitting the exam” and the MW students are therefore well prepared for the exam, how do you explain the dismal pass rate? I know of no other exam system where the pass rates are so poor. What more evidence do you need than that?

    It is insufficient to merely suggest, as you do, that the exam is difficult. There are many subjects in the world far more difficult than wine where pass rates are substantially higher than those delivered by the IMW. This in itself is evidence of a failing system.

    The comments I made re the institute being “old fashioned” and stuck in the “dark ages” are references to the fact that the entire British education system back in the 1950’s was as elitist then as the IMW is now. Thankfully, much as changed and the school and university system is now inclusive rather than elitist. This change happened in the face of cries and complaints from the more privileged members of British society that benefited under the old system. British society is better as a result.

    Sadly the IMW appears to have changed very little over this time. Consequently the IMW has failed in its original brief, as stated on the IMW website, which was “to improve the standard of education in the British wine trade”.

    Unsuitable or offensive? Report this comment

  • I did not realised that you were only trying to spark debate, otherwise I would not have commented. I wanted to help you as I thought you wanted to report something meaningful about the IMW and just went down the wrong path.

    The fact that I am the only one who has ever commented on something you have written should tell you a lot about the effectiveness of your writing style.

    Sincerely, KB

    Unsuitable or offensive? Report this comment

  • I am delighted to count your comments as one of many comments I have received on the various pieces I have written for Harpers, not just the on the blog but articles too. But much depends on the controversy of the subject matter and how raw the nerves are.
    Much of my writing is tongue in cheek.
    In your case the penny has presumably dropped as to your real chances of success hence your reaction. Thank you for the debate and let's hope you are in the tiny group that may have a slight chance of passing the MW. I am sure your robust defense will have earned you some brownie points from the powers that be. Good luck, it's been fun!

    Unsuitable or offensive? Report this comment

  • There is no question the MW exam is extremely difficult. Most members of the trade would say that Master of Wine and Master Sommelier are the 2 most prestigious credentials in the wine world. In fact I can promise that the passing rate is well below 10% for both of them. I have many friends who attempted both exams and stopped short, many who continue to try and many who have succeeded. I passed the MS in 2004 and have not considered attempting MW due to the time commitment involved. I do respect the IMW however & have tremendous admiration for the 4 people who have passed both MS & MW. Both organizations suffer from a perception of elitism but they are both nearly impossible exams that if you are one of the few to pass it is recognition for all of the work you have done and all of the skills you have developed over many years. Neither organization is a school. They are both credentialing bodies. I have witnessed both of them continuously evolve over the last few decades. Educational practices have improved as has transparency. Regardless of whether someone passes either exam, they will have learned a tremendous amount in the process. The wine and restaurant trade are improved by both of these organizations by inspiring countless passionate people to study, taste and travel relentlessly in their pursuit to climb the Mount Everest of the wine world.

    Unsuitable or offensive? Report this comment

  • You make some good points David, thanks for your response.
    The question is “Is wine really that difficult a subject?” or “is it being portrayed as something much more difficult than it really is?”
    We’re not talking about Quantum Physics or Nanotechnology here; we’re talking about the simple subject of wine. One could equally create a difficult exam that almost nobody could pass on the subject of “The life and times of Teletubbies”, but what would be the point?
    If the IMW came clean with the fact it is an exclusive club rather than a credentialing body I would have no real issue. Clubs can create whatever rules of entry they like as long as they adhere to the laws of the land. But as a credentialing body it needs to be fair and reasonable to be deemed relevant. I don’t think it passes this test as I argued in my blog.
    The purpose of an exam is to test candidates’ knowledge of the subject, not to test if they are able to decrypt vague ambiguous exam questions. It is on this point that the IMW fails, precisely because it appears they lack people whom really understand this point.
    The fact that no one from the IMW has entered the debate would suggest acceptance of this argument.

    Unsuitable or offensive? Report this comment

  • Stephen

    I would take issue with your assertion that wine is 'a simple subject'. Like any field, be it quantum physics, nanotechnology or winemaking, it is not simple. Wine is a combination of art & science - it's as much about gut feel by an experienced winemaker as it is the chemical analysis in the lab.

    Whilst I can understand criticism of the IMW, take a look at an Oxford or Cambridge History paper - they are very similar questions - on initial assessment quite simple, upon further analysis some can be infinitely complex.

    There's a reason people devote their entire working lives to particular regions - ask Jasper Morris if he thinks Burgundy is simple.

    One of the joys of working in wine is that every year it's different and every day you learn something new. The MW exams are deliberately hard, just as the MS exams are - and you can't say that people who go through any stage of the process don't benefit from the experience, even if they fail.

    Unsuitable or offensive? Report this comment

  • Hi Tom,

    Good arguments! I certainly agree with your point that people going through the process will benefit from the experience. No issue on that one.

    Where I disagree, is with your comparison with the Oxford or Cambridge History papers.

    The pass rates are, according to my research, for these exams are around 60%.

    Compare that to the MW of “well below 10%” according to David’s comment.

    How do you explain that?

    Unsuitable or offensive? Report this comment

  • Stephen, your article is spot on. The MW examinations are structured to allow the existing IMW power structure to gatekeep the club, pure and simple. Yes, wine is a complex phenomenon, but it is also one where both knowledge and tasting experience are fairly easily and readily amassed by many people. I am a retired lawyer and veteran of four bar examinations during my career. Law school and bar examination essay questions follow a pattern similar to the MW exams (and Oxford/Cambridge history papers, apparently), but with a key difference in the case of the MW exams: each bar question typically contains multiple clues which are calculated to elicit relevant responses from those astute enough to pick up on most or all of the clues. Quite often, there is no right or wrong answer, and issue-relevant discussion will carry the day. What is sought is the ability to "spot the issues", as law professors like to say, and to discuss them intelligently, articulately and as thoroughly as possible under considerable time pressure. Legal relevance, often cutting-edge, and level-playing-field fairness via the clues are the name of the game. One will find no "problems inherent in organic orange winemaking by Somali refugees in Croatia" issue-spotting. And therein lies the difference. The MW evaluation process is arbitrary and too often totally irrelevant.

    The few defenders of the IMW (strangely, more often failed MW candidates or friends and relatives of failed or struggling MW candidates, rather than actual MWs...I assume that club rules forbid engaging the hoi polloi in a debate upon the merits of the IMW) go on and on about how impossibly difficult the MW process is. They often point to legal, medical, accounting and other true professional certification processes, and observe that some total incompetents are allowed in, so therefore, the processes must not be nearly as hard as the MW program. What rubbish! The truth is that, historically, the professional gauntlets are much more difficult and time-consuming to run, much more expensive and run by candidates of significantly greater average intelligence in the first place. However, the goal is not admission to a relic akin to a British old boy's club. The goal is to insure, as best the relevant authorities can, the competence of professionals who have critical, sometimes life-and-death, impact on society. Measure the number of MWs who have contributed absolutely nothing to the world of wine (unless as wine store clarks with MWs after their names) versus the total number of MWs to the number of brain surgeons who have lost their licenses versus the total number of licensed brain surgeons. Go down the list of MWs and pick out the names of MWs that you know. (Maybe 1%, but only if you are British!) I did that analysis, and I think that I came up with about a dozen, with only Jancis, Belfrage and Lewin having ever written anything worth reading. (Remington Norman famously quit the club, and others have apparently quit rather than be bothered to pay dues and attend club events.) You drop from there to the Asia-based, "my husband is an investment banker!" set. Jancis surely dissed the MW program vis a vis the MS program, and when one considers the bad press generated by the most famous MW, and the fact that number two (Remington Norman) quit and apparently would not discuss the club, one learns what one needs to know about the IMW...

    Unsuitable or offensive? Report this comment

  • Stephen

    The MW paper, by necessity of the product it is address, is asking questions about an extremely broad subject - so you might get theoretical questions ranging from soil types in Alsace to commercial production of Californian Cabernet Sauvignon.

    Continuing with the historical analogy, it would be like sitting a history paper and not knowing whether you're going to get a question about Genghis Khan or the American Civil War.

    I'm looking at the 2014 questions now. Upon re-reading your article, you are proposing that the examiners want you to answer a different question, which is just untrue and not what Jancis said at all. What the examiners are giving you is a very broad arena in which to answer the question. For example, in this years Theory of Wine: Contemporary Issues, one of the questions is:

    2) Does the wine industry lack innovation?

    Where to start! You could convincingly write in detail about tracking technology, shipping and logistics, electronic trading, the Bordeaux Place, use of the internet, promotions, marketing, branding, social media and a myriad of other topics.

    I suspect that many people answer the questions thinking initially that they will have a lot to write about, not doing an essay plan, and suddenly discovering that the answer they've spent 25 minutes on is actually quite thin.

    Then there's the blind tasting, which is phenomenally hard and the part that I understand most people fail on. Looking at the 2014 wines again, amongst them you're asked to blind taste Syrah from Australia, California, Chile & France!

    Looking at the wines and the questions, it's not entirely surprising that the pass rate is low. Oh and let's not forget that you have to do a dissertation too.

    Unsuitable or offensive? Report this comment

  • I think Bill Klapp answers the point very well, and his comments pretty much address the remarks you make Tom, but to add my tuppence -- the key point here is whether the exam open question permits an open answer as long as it is well argued and relevant, which is fair and reasonable, or whether the open question is looking for a detailed specific answer, which is the case made by Jancis Robinson and the associated videos, in which the candidate has to second guess the examiner’s mind – this is widely accepted as a poor quality exam question.

    The difference between a good exam question and a bad exam question is that a good question is a test of the candidates’ knowledge. Whereas, a bad question is a test of whether or not the candidate can decrypt the question and is focused more on that point rather than testing the candidate’s knowledge of the subject.

    I’m beginning to sound like a repetitive old buffer so unless someone can really come up with an alternative good reason why the pass rate is so appalling I will refrain from further comments.

    Thanks to all those that have contributed in this debate. I’m sure we could continue to the point where we crash the entire Harper’s IT server, (and Mr Siddle will never forgive me), so on that note I will sign-off and I wish you all well.

    Unsuitable or offensive? Report this comment

  • Tom,

    You chose an interesting example. You use the Gengis Khan/American Civil War analogy, but that, in and of itself, may make last-minute preparation impossible (not unlike the situation of contestants in general knowledge-based TV quiz shows), but it does not necessarily make the history paper difficult, as most earnest history majors should have something to say about both subjects. The specific wine industry question calls for an opinion, not true knowledge, but surely solicits a far-ranging discussion of the topic. It seems to me that your apparent top-of-head listing of potential discussion points would put you well on the way toward a high-pass score on the question, so it becomes only a matter of how well you organize and write. However, it would be difficult, if not impossible, to grade the answers in any reasonably objective fashion. The question could be answered with dozens of other questions: innovation...where? The U.S.? The U.K.? Hong Kong? Paris? That game could go on forever. (Of course, answering with answers would seem a sure course to failure.) The key to the type of professional examination that I describe above is that the examiners are compelled to generate model answers, and to rate the examinees against that standard. I could be wrong, but it is not clear to me that that approach is always taken at IMW. If it were, then we would have only two factors at play: either there should be far more MWs than there are, or the overwhelming majority of the candidates simply do not possess the right stuff to succeed. There is more than a little evidence that the latter is the case in any event, and that often extends to tasting and writing abilities exhibited after the magic letters are affixed to the names of successful candidates. (The rather undistinguished track record of MWs as tasters also calls into question the relevance and value of the MW blind tasting exercise, save as an exclusionary tool.) I am sure that neither Stephen nor I will ever amass the information required to prove conclusively that the MWs are but an exclusive club, but the available factual and anecdotal information strongly suggests that is so...

    Unsuitable or offensive? Report this comment

  • A final thought: the Pancho Campo affair...how he was admitted in the first place and how he found his way to the door...would justify the sort of scrutiny that Stephen has offered here.

    Unsuitable or offensive? Report this comment

  • Just re-reading the comments above. I love David’s comment on “perception” here:

    “I do respect the IMW however & have tremendous admiration for the 4 people who have passed both MS & MW. Both organizations suffer from a perception of elitism but they are both nearly impossible exams”

    Honestly, you couldn't make this stuff up if you tried. :-))

    Unsuitable or offensive? Report this comment

  • If I remember well, Pancho Campo resigned and the IMW even wished him success in his future activities. I never saw any indication on the IMW's side that there was anything they reproached themselves with about Mr Campo's admission.

    Unsuitable or offensive? Report this comment

  • One thing that I will say about the MW is that the ones I've known have at least been well educated and interesting people. The MS is utterly ridiculous with some of the most shallow people whose education never went beyond high school or some irrelevant 4th tier regional state university conspicuously trying to overcompensate through their wine certification. Also, the MWs at least need to know how to reason out a proper essay question and present the response in well written English.

    As for the alleged difficulty of the MS test, I could create a certificate program, let's call it the MBS (Master of Baseball Statistics). As for the test, even though it doesn't contain any significant essay questions nor call for any true analysis and reasoning, I could rig it with such esoteric questions that less than 10% would pass. At the end of the day, however, it is still nothing more than a test of the rote memorization of baseball statistics. The notion that the pass rate is somehow indicative of the subject difficulty or academic rigor is foolish. A doctoral program in physics at MIT or in history from the University of Chicago has a much higher success rate than 10%, yet only a fool (or a poorly educated sommelier) would ever contend that they are the same thing, much less somehow easier, as memorizing wine facts.

    Unsuitable or offensive? Report this comment

  • Only the Brits could have come up with such a ludicrous title as "Master of Wine". You do not master wine, it masters you.

    Bill Klapp, they miss you at Berserkers.

    Unsuitable or offensive? Report this comment

  • The students sitting the exam know the depth to which the IMW wants the question answered........... OR THEY SHOULD NOT BE SITTING THE EXAM......... end rant

    So yes the questions look easy, but you have many hours to fill in the details, don't know enough details ...... you fail.

    So what was the question posed?

    Unsuitable or offensive? Report this comment

View results 10per page | 20per page | 50per page

Have your say

These comments have not been moderated.

You are encouraged to participate with comments that are relevant to our news stories. You should not post comments that are abusive, threatening, defamatory, misleading or invasive of privacy. For the full terms and conditions for commenting see clause 7 of our Terms and Conditions "Participating in Online Communities". These terms may be updated from time to time, so please read them before posting a comment.

Any comment that violates these terms may be removed in its entirety as we do not edit comments.

If you wish to complain about a comment please use the "report this post" button or email harpers.editorial@wrbm.com.

Mandatory
Mandatory
Mandatory
Mandatory
Twitter Facebook YouTube Linkedin