2019 It’s All Been Written Judging Process

All stories were submitted to an email that only network head Jerome Wetzel has access to. The person writing this is Jerome Wetzel, so I’m going to switch to first-person pronouns.

I downloaded each entry as they came in. If the file didn’t have an author name already in it, I didn’t add one and did my best to forget whose entry belongs to whose before I read them. This was pretty successful, as for the most part, I had forgotten. This arrangement was also disclosed in the official contest rules, posted at the time entries were first accepted, with myself listed as both the receiver of the stories and one of several judges.

Once the contest was closed, I then read every single entry all the way through. This is the part of the process where bias can most enter in, as some files did have author names on them. I believe I am an objective judge, and asking a few people who had interest in this competition (not necessarily entrants), they agreed. But I realize that no one can ever be 100% unbiased, and if the network had a bigger staff, I would have removed myself from this part of the process as well (more on this below).

I ranked all entries from first to last in my estimation based on the quality of the story, doing my best to ignore grammar, punctuation, and other minor issues.

Judging Panel

This year, having so many entries, I sent the panel of five pre-arranged judges (excluding myself) what I judged to be the Top Ten entries. The panel included:

  • Alanna Gibson, Ensemble member at MadLab theatre, model, dancer, and actress

  • Kristin Green, Music Director / Actor in It’s All Been Done Radio Hour and band member of So Long, Stargazer

  • Nikki Smith, Managing Director of MadLab theatre and star of Nikki Tells You How to Live Your Life

  • Laura Spires, Artistic Director of MadLab theatre

  • Seamus Talty, Foley artist for It’s All Been Done Radio Hour

The version of the stories the panel members received included no author names, as author names had been removed from files where they existed. No list of authors was sent, either, so the judges had no idea which ten authors they were reading, and I did not reveal this information to them. Even if they knew someone had entered, they did not know if that person was included in the ten stories they received. This is the completely blind portion of the judging.

Each judge read all ten stories and ranked them first to tenth in quality.

Aggregation

Once all five judges had turned in their rankings of the Top Ten stories, I entered them into a spreadsheet, assigning 1 point for their first place finisher, 2 for second place, and so on. I also added my point totals for the order I had ranked the Top Ten. All six judges, myself and the panel, had equal weight, with Excel providing average scores based on the six point totals for each story. If I had bias in my ranking, it would still only account for a minority portion of the total score. In this particular case, this year’s grand-prize winner would still have had the top score with my rankings removed from consideration.

The grand-prize winner was named based solely on the stories’ numerical score. Additional winners were next five lowest scores, with the decision made to cap the number of winning stories at six because there was a noticeable gap in scores between the sixth and seventh scores, and through consultation with the judges as to how many winners should be named from the pool of ten they read.

Ensuring Fairness

Given that an author who works for the network was being named the grand-prize winner, I felt it extra important to provide as much transparency to the process as I could without betraying the anonymity of judging scores. To this end, extra steps were taken to check my work in compiling final scores that were not planned ahead of time. These steps increase the reliability of the winning rankings, and so will be repeated in the future.

The score spreadsheet used to determine the winners was sent in a single email to all five members of the judging panel. The judges’ names were removed from their score columns, but each judge could see that their scores were included and that the total weighting and ranking was fair and equal for each judge. With all six of us on the email chain and able to see that everyone was receiving the same spreadsheet, judges were encouraged to reply all if they did not see their score column or found anything not above board about the totals. If a judge had done so, the rankings would have been invalidated and the contest would need to be re-judged. This did not occur, as the rankings were accurate.

Also, this page was created to outline these steps and answer any concerns anyone might have as to how one of our own could win the competition, and give confidence to future entrants that they will be judged only on the merits of their story and not on who they do or do not know.

Future Competitions

In a self-review of our process, I am confident that enough steps were taken to ensure the right story won and bias did not sway the results. However, I also want to do everything I can to improve our process in the future. As such, an email has been set up - judging@iabdpresents.com - where suggestions can be sent for further refinement.

I also pledge to name an individual who is not judging the stories next year to receive the entries and remove identifying information, such as author, from them, so not a single one of the judges will have access to author information until decisions are made.

Lastly, a multi-round judging process will be put in place before the 2020 contest to ensure at least two judges review every single story, and that the finalists (whether they be Top Ten or another number) are chosen by more than a single individual. This year, we received more entries than I had told the judges they would need to read, hence the winnowing down to a Top Ten for the full panel. Next year, judges will be asked to read more stories. But rather than having every judge read every story, a major undertaking, we will batch them, have different individuals review each batch (with each entry being in at least two batches), and then put the winners of each batch into a pool for final judging.

Last Notes

I hope this helps you understand our process and gives you confidence in it. If there are still concerns, please send them to judging@iabdpresents.com and we will be happy to respond.