Recently I’ve been fortunate enough to be a reviewer for a couple of conferences: Agile2011 and the Seattle Scrum Gathering. It was an eye opening experience for someone like me who has been on the submitting side of the process to see and participate in how the reviews actually get made.
As a rookie reviewer, I wanted to share some of my experiences with the review process in the hope that others may learn from it. I do want to emphasize that overall, it was a wonderful experience that I enjoyed and hope to do more of in the future. That said, I also wanted to be unflinching in my reflections on what could be improved in my own performance.
I’ve been too self-conscious in my feedback – afraid to take a stand. As a reviewer, it’s my job to have an opinion and not water it down in the interests of appearing “balanced”. Balance will come from the other reviewers, not from me.
I’ve been a little snarky or rude. In general, I do not do this too much, but everyone has a bad day. As someone who goes through the proposal process as a submitter, I know how painful it can be to be flamed by some jerk reviewer having a bad day. As reviewers, we need to keep a lid on that. We have an obligation to be firm in how we express our opinion, but polite.
I’ve been too brief in my feedback. Sometimes just asking for more detail is not enough. When I’m lazy, I can just blow through reviews and simply ask for more detail without expanding on anything else. I’m coming to realize that’s a cop-out. I was provoked by someone once to provide more detail. Their proposal was actually fantastic and I didn’t have a whole lot to say about it other than, “Great job!” Well, rightly enough, they called me on that and asked if I could provide a little more feedback that is useful. I got a little angry with myself, sat down, and pounded out a critique that was nearly two pages long. I was surprised with myself. It was good. Afterward I had to ask myself, “Why don’t I give everyone that kind of quality feedback?”
Not understanding the audience for the feedback. It’s one thing to communicate with a private audience of reviewers, “Hey, this proposal is total crap. The guy didn’t even bother to give his full name…” it’s an altogether different thing to share that feedback with the submitter, “Dear so & so, thank you for your submission, while there were many great proposals, we couldn’t accept all of them. Yours was rejected due the brevity of its content.” You don’t want to get stuck re-writing the feedback for hundreds of reviews. Even more reason to write reviews really well the first time.
Fortunately the system is pretty robust and forgiving of error. Given that there are teams of reviewers who provide multiple reviews, it allows room for error (and learning!). I don’t have to provide stellar feedback for every review if there are 5 other reviewers also providing feedback – hopefully one of us is going to provide some valuable input.
For me, the conference submission and review process has been about learning. I’ve learned as someone who submits proposals that sometimes even the best material will go overlooked by a given set of reviewers for conference. I don’t take it too personally. There is a rich ecosystem of speaking opportunities available to me as a speaker, and if I can’t find an opportunity one place, then it’s very likely that I will find it in another. If I am passionate about the topic, I am usually pretty darn persistent about it too. This persistence has paid off. The more I get to know some of the more prominent speakers in our business, the more I realize and respect that they have been dealing with these sorts of vicissitudes in the process for many years.
As a reviewer, I have been lucky enough to participate in making the sausage that we call a review. It is a messy process with lots of flaws. I feel an obligation to improve my reviewing skills over time in order to provide folks who submit proposals the best possible experience with the proposal review process. But as groups separated by a screen of sorts, it’s easy to get frustrated with the process.