Reviewing Conference Proposals is Like a Job Interview
I just got through my first experience as a reviewer for a couple of conferences and I feel like I learned a lot in the process. I made a lot of mistakes, some of which felt silly and others I still feel a bit bad about. You see I have been submitting proposals to conferences myself for a couple of years, and I know how heartbreaking it can be to get your proposal turned down for a conference. Especially when you know your material is really great. So, it was both revealing and initially a little bit scary to be on the other side of the process.
To begin with, often all you have to work with is a submission form from an automated system. This really severely constrains the amount of information that you have about a given proposal or the person who submitted it. It reminds me a lot of the job interview process. As the hiring manager, all you get for your initial input is a resume – perhaps the world’s most ineffective information communication tool. Somehow, using only the text on the page, you have to divine the personality of the applicant, their knowledge of the subject domain, and assess the overall merit of their application.
Now, if your experience with the job interview process has been anything like mine, you know all too painfully well that there is almost no way in hell to choose a decent job candidate solely based on their resume. The information is excessively sparse, there is no feedback, and you have no way to validate the assertions that are made. Oh, and you have many more candidates than you have jobs, so contrary to what they might tell you in HR, you are very likely looking to filter people out.
So when you are filtering resumes, desperately trying to find the good candidates, you usually adopt some criteria for assessing the quality of their resume. These criteria are usually things like:
Spelling and Grammar
Clarity of thought and presentation
Attention getting words, thoughts or ideas
Relevant experience
Etc.
Of course, none of these criteria really translate into a guarantee of a superstar future employee, right? In fact, all of those criteria are pretty weak indicators of quality overall when you are looking for the next great programmer. However, initially they are really the only guides to you are given to assess whether or not a candidate is worth investigating further.
The same problem applies to reviewing submitted proposals to a typical conference. You don’t have anywhere near enough information, and the criteria that you apply are very likely inadequate to the task of identifying a quality proposal. Therefore, you end up with a set of criteria like this:
Spelling and Grammar
Clarity of thought and Presentation
Attention Getting words…
Relevant experience…
I hope that you can see where I’m going with this by now. It is an imperfect system at best. It can be further aggravated by submission systems that actually conceal information in the interests of fairness. Some conferences will “anonymize” the proposals so that you do not actually know who the submitter is. I think this is done in the interest of creating reviews that focus on the merit of the ideas alone and not the reputation of the presenter. This practice has an unintended consequence of further restricting the information that the reviewer has to work with. Imagine reviewing resumes where you cannot see names or work experience and you start to get a feeling for what working with anonymous proposals might be like.
Details
At its most basic, with some review systems I feel like you are really left with the following: Did the submitter care enough to provide a detailed description of the proposal and how it would be presented? Were they willing to invest the time and effort to provide me with as much information as possible? My experience is that all too often people, even very experienced presenters, will skip over entire sections of the submission form or provide only single sentence answers. Often, you can very quickly break the pile down into two stacks: people who bothered to fill in all the blanks with some decent detail and those who do not. I think many folks who submit to conferences would be stunned to see just how often people neglect to fill in the details.
As a reviewer, you are left with two stacks: those who did provide detail and those who did not. Which stack would you prefer? Now does that mean that people who left out information in the proposal had poorer presentations? No. It is very likely that there are some great proposals that get overlooked this way – in fact, I’m quite sure this happens all the time. However, let’s face it, getting your submission accepted to a conference is a competition. You need to do everything you can to understand what the reviewers are looking for. First, I can tell you that rich detail sells big. It tells a reviewer that you are willing to do the extra work to sell them on your proposal. Investing in the detail suggests that you may understand your topic and know how to deliver it. Even with rich detail, there is no guarantee that the presenter is any good, but what else do you have to work with? Often not very much.
I have seen especially impressive proposals where people provide links to video of themselves giving the presentation, links to the PowerPoint slides, and more. When someone is able to put additional material like this into a proposal, I find it very impressive. It tells me that they are very passionate about their topic and that they are willing to go out of their way to provide additional detail (that reviewers are starved for) in order to be considered. Very few people bother to do this, so when people actually bother to provide this kind of information, they *really* stand out. It is still not a guarantee you will be accepted, but believe me it puts you closer to the top of the list than the bottom. Just like in job hunting, you want to do anything you can to make yourself stand out.
The Problem with Themes
Even filling in all the blanks and providing significant detail often isn’t enough. I think it guarantees you are in the hunt, but there is more to consider. One of the toughest considerations as a reviewer is theme. Often there is some sort of conference or stage theme that you are responsible for satisfying as a reviewer. All too often I have seen terrific proposals that I was convinced would make compelling and interesting sessions, rejected because they didn’t appear to match the theme of the conference or the stage they were submitted to. For example, if you submit a proposal on “Writing Great User Stories” to a conference that has “Radical New Ideas” as a central theme, you are more than likely going to be rejected. No matter how great the material is, no matter how wonderful a presenter you are. If there is a perceived mismatch between your topic and the conference or stage theme, you are very likely out of the running.
Now I think that by their nature, themes are dreadfully subjective and vague and this is a bit of a tough nut for the submitter and reviewer to crack. I think conference organizers feel compelled to use themes to help give their attendees some sense of the value they intend to provide. It seems that most conference organizers do not feel compelled to just fill their agendas with any old good presentation that comes along. They also do not want repetition. From what little I’ve seen so far, it’s pretty easy to end up with three different proposals that all seem to boil down to, “Another Intro to Scrum”. Even if you have great presenters like: Mike Cohn, Jeff Sutherland, and Ken Schwaber – in the end, only one gets picked. In addition, if it is a Lean/Kanban conference, probably none of them gets picked. That is regardless of the quality of their stellar proposals or their godlike presentation skills.
Is that fair? I don’t know. Personally, I hate it when I feel I have material that is valuable, has been well received by my audiences, and supported by a solid proposal – and it is rejected. I am a competent presenter. I want to tell the reviewers what boneheads they are, that they missed a great opportunity! I want them to know they could not see value if it kissed them on the nose! But I don’t. That just does not seem like a very smart approach to me. So, I have a beer with my friends and go on a bit of a rant – fortunately they tolerate me, and I get over it.
Decision Time
Themes aside, sometimes it also just comes down to a matter of taste. As a reviewer, you are confronted with two great proposals and you only have room for one. You have to make a difficult choice where there is no obvious winner. In a case like that, it really will come down to some sort of gut instinct (often wrong) that you end up relying on to make the choice. You can put in place scoring systems and other mechanisms to make the decision appear more objective, but the bottom line is that it is a subjective judgment and you have to make a call. Of course, it does not make you feel any better when you are on the wrong end of the decision. It is hard to understand how your rocking proposal that you poured your heart and soul into could have been rejected.
Those feelings are natural enough and I understand all too well how they come about. The point is that you need to keep the bigger picture in mind. A rejection by a given conference may have little or nothing to do with your skill as a presenter or your mastery of the subject matter. That’s part of what makes this process, like job hunting, so bloody frustrating. We would all like to crack the code and have our genius recognized. However, the process, the information, and the people are imperfect. There are a depressing number of ways that great material can be overlooked.
If I haven’t put you completely off your feed by now, I’d recommend a couple of resources to follow up on if you are interested in improving your chances of getting your proposals accepted:
Mark Levinsons Blog:
http://agilepainrelief.com/notesfromatooluser/2011/04/how-we-reviewedagile-2011-coaching-stage.html
Mitch Lacey’s blog:
Part 1: http://mitchlacey.com/Blog/Agile-Conference-A-Stage-Producers-Story-Part-1.html
Part 2: http://mitchlacey.com/Blog/Getting-your-Session-Accepted-to-the-Agile-Alliance-Agile-Conference.html
Part 3: http://mitchlacey.com/Blog/Stage-Producer-101-Building-the-Best-Stage-Possible.html
#submissions #reviewers #Presentations #reviews #conferences #proposals #themes