How Are the Exams Graded?

Some helpful exam insight from Kim B [TM]

You're frustrated. You've slaved over your exam, looked EVERYTHING up in the Style Guide and dictionary, but you still didn't pass. "I followed the SG to a T!!" "Is there a glitch in the exam?!?" "HOW DOES A COMPUTER KNOW HOW TO GRADE ANYWAY?!?!"

Well, I'll share the secret of how the exams are made and graded. It's not really a secret, per se; I've mentioned it here and there from time to time, but I figured I'd make it its own post.

Please note that this is my personal exam-making process. I've made and/or monitor most of the active English-language exams on TranscribeMe, but not all.

1

An Audio is Selected — Pretty straightforward here; we select an audio that we feel will adequately test the skills we are assessing for a given exam.
Hint: the keys to a stellar transcript are accuracy (get the words right), research (don't be afraid to google), style (know the SG), and readability (you don't have to be an expert on English grammar, but please punctuate well).

2

The Audio is Trimmed — The audio is then sliced and diced to be a reasonable length (around 4 minutes), and "subjective" bits are removed from the audio (at least, for audios I make), i.e., anything that could result in too many interpretations and therefore require more exam keys.

3

Keys Are Made — I then make keys for the audios, including all *legitimate* variations I can think of, whether it be differing stylistic interpretations or a difficult part of the audio that could be (understandably) heard different ways. Some audios have over 60 keys! Others have less than 5.

4

The Audios Are Tested — We then get real people, just like you, from our platform to give the audios a go. We carefully examine their results against the keys, adjust the keys where necessary ("Oops, I missed/misheard a word? Better fix that." "Oh, this person did it this way, and that could be correct too. I'll add that as a variation in the keys.") and then set the pass rates.

5

Setting the Pass Rates — To set the pass rates, we examine our transcribers' submitted results and "QA" them a bit. This phase is where we allow leniency for variations that cannot be accounted for in the keys or minor errors that would not alter the accuracy of the transcript as a whole. This is where some audios will have a significant discrepancy in their scores, usually depending upon accents or audio quality. Oftentimes, however, something as simple as false starts or crutch words could be your downfall. Make sure you review the False Start and Crutch Words portions of the Style Guide before attempting your exams!

6

Grading the Exams — When you hit that Next button on your exam, your results are then submitted into our system which matches your transcript against all the beautiful keys, determines the closest match, and grades your accuracy against that match. This is very similar to the change percent you see on your QAed jobs. Some exams are graded a bit differently depending on what the exam is testing (e.g., the QuickStep and QA Exams), but that's the basic gist of it.

7

Monitoring the Exams — Then, I keep an eye on the exams. I have a spreadsheet where I paste in our Exam Report that tells the scores earned for each audio. I can then analyse these results and audit/tweak the pass rates and keys as needed. Maybe there's an audio that's so tough no one is passing it, so I'll just deactivate it. Maybe an audio has a lower-than-average success rate, so I'll lower the pass rate a bit or add more variation to the keys. Or, vice versa, if the success rate seems unrealistically high (as in, hardly anyone fails it), I'll raise the bar (which is why some audios have over a 98% pass rate).

Welp, that's it in a nutshell! Hope that helps alleviate any frustration a bit; rest assured that we take every possible measure to give everyone the opportunity for success on their exams.