December 2, 2020

There’s always a Reviewer 3…


If stopped on the street and asked for the opposite of success, many passers-by would respond with “failure.” This makes sense, these terms describe the two opposing ideas in a simple manner, quickly conveyed to others. Unfortunately, this simple manner does not convey the most valuable information. Success and failure are not independent of each other, rather success requires failure, and failure requires success. In order to measure success or failure, expert judgements are made and critical feedback is given. Then you, and only you, decide if this instance was a success or failure.

As a graduate student and early-career researcher I would have acknowledged my professors’ grading, my employer’s performance reviews, and my declined grants as criticism, not feedback. The hardest learning experience, and greatest moment of clarity, involved the transition in viewing criticism as feedback. Using a few academic studies and my first grant writing experiences as examples, I’d like to show you what this process looked like for me, and talk about what I learned from the process. Through this example we can examine how peer support, perception, and previous experiences can influence the transition from criticism to feedback. Finally, we’ll talk about the reason these are failures, and how redeeming successes within each example far outweigh the single failure.

A 1999 study by Kamins & Dweck examined student responses to failure at very young ages. Within this study, two patterns formed in student reactions. They described the first pattern as the “Helpless pattern” of reaction to failure. Considering how I felt in grade school on test day, I imagine we all understand the feeling of being helpless. Within the context of the Kamins & Dweck study, this pattern saw failures within the study as personal failures and a measure of ability. Negative thoughts (e.g., “I’m not smart enough for this!”) could form more easily, and “impaired strategies and performance” followed. The second pattern, a “Mastery-oriented pattern,” focused on the methodology used to achieve study goals. By focusing on the methodology as the key failure, students should receive encouragement to learn from the example and apply new methodologies. In addition, students can maintain expectations of future success, positive self-assessment, and constructive behavior despite setbacks because they did not focus on personal failures to achieve study goals.

To be quite honest, learning to move out of the “Helpless pattern” in my own research took some big failures (including getting kicked out of a graduate school program), and a group of colleagues who laughed at me and were able to explain, in spite of my emotional meltdown, that this was a normal setback. By explaining their own setbacks, they demonstrated another influential element of receiving feedback, understanding the interpretation and application of feedback.

In 2001 Hyland & Hyland examined the influence of teachers on student responses to perceived failure. Two teachers in English as a Second Language (ESL) courses participated in the study, with two different methods of giving feedback to students. Though both teachers used written feedback, seeing the method as more “informational” to students whose verbal proficiency is still developing. One teacher gave feedback via a “point for point” feedback sheet. Through creating “suggestions” tied to specific critiques, both strengths and weaknesses receive independent examination, and improvement goals added to each. In contrast, the other ESL teacher used a global feedback method, in which she applied comments directly on student essays. This global feedback method encouraged broad trends to surface in individual student’s work, and feedback could be used to encourage general improvement with “soft” suggestions to focus this improvement.

I enjoy the example of an ESL class, as grant writing was a very new language for me last year. I submitted my first grant during the first week of August, 2018. By early November, this grant received a re-write for a second grant solicitation, however the first set of reviews were nowhere to be seen. I imagine, though I possess limited grant writing experience in grant writing, that nobody in academia really forgets their first grant review. Mine came in December 2018, had four reviewers, and required a PhD to translate for me at the time. Unfortunately, there was no check attached to the message. There was, however, a valuable lesson on failure vs feedback.

Each reviewer, though anonymous, had a different personality in their reviews. This personality emerged in the language within each review, both word choice and phrasing. Reviewer 1 (R1), for example, broke down the criteria for judgement much like the second teacher in the Hyland & Hyland study. Alternating the strengths and weaknesses displayed in our grant as they read, the organization of strengths and weaknesses gave a broad judgement of the grant as a whole when examined against individual criteria. This judgement on our grant felt validated when the reviewer, asked to give a summary of our proposal, showed a good understanding of the grant. While several details in our grant needed clarification, our research team had gotten the message across, been seen by an actual human (I hope), and R1’s feedback was specific and constructive. I personally enjoyed the idea of an academic reading our grant and taking notes, in the same manner as a student in a library, showing in the flow of the commentary. Reviewer 2 (R2) used bullet points to separate strengths and weaknesses, always leading with the strengths. Likewise, their summary let our research team know the message of our research, if not the details, were communicated decently.

And then the bomb dropped.

Reviewer 3 (R3) can best be summarized with, “DATA MANAGEMENT PLAN: Sufficiently described.” This was, by the way, the only comment in the entire review not overtly negative. Each section of the 235-word review began with, “The proposed project does not…” When asked for a summary of their opinion, R3 was very blunt in stating, “The proposed project will do little to advance,” the goals of the program providing the grant. Ouch. I couldn’t bring myself to read the fourth review at the time and let the helpless response pattern cloud my judgement. As I would learn, however, the failure I experienced in this case was my poor interpretation of the first 3 reviews.

I read the first three reviews before consulting my fellow research team members. With our primary investigator living in Connecticut, and the other member of the senior research team unavailable until the following week, I had a great deal of time to overthink the issue. Five days later, our senior research team read the reviews together. I mentioned the anxiety of R3, and my inability to read Reviewer 4’s (R4) comments at the time. We talked about the reviews and what useful comments we could find. But, through the help of my team members, both experienced PhD’s with research backgrounds, I saw each comment in a new light. For the first time I read R4’s comments. As I mentioned earlier, the language chosen by the reviewer could add personality to their comments. The word choices and language of R4 forced me to reinterpret the first three reviews. Strengths received light praise, but the importance of these strengths to the overall project had additional comments and suggestions for implementation. Weaknesses received critical review in unambiguous language. Each of the weaknesses came with suggestions for improvement; each beginning, “A stronger proposal would….”. This language offered improvements without implying a lack of strength, validating project design choices while giving constructive criticism. Commentary such as, “This reader wonders…” showed thought and reflection on our project by the reviewer, and interest in our grant proposal. Finally, the summary requested similar specificity changes to R1 and R2. To my great relief, however, we clearly communicated our project through the grant request, described the importance of the subject realized, and confirmed the necessity of our research.

This grant proposal, objectively, was a failure. R3 made sure to emphasize this with automaton-like clarity. Our failure to receive funding, however, encouraged a closer examination of future failures. When our second grant returned the next spring, also unfunded, with a far more critical set of reviews, I was ready. I took the weekend to study the responses, summarize them as best I could, and compared them to the previous grant reviews looking for common elements. I didn’t focus on individual reviewer criticisms as personal failures. Instead, I focused on comparing all of the reviewers across both grants for big ideas. I entered our team meeting with a plan, gave and took feedback on the writing process, and made plans to continue searching for future grants. A few weeks later, I found out my application to my PhD program of choice was denied. Another failure, another learning experience. I kept on going, because I failed before and will fail again, but I will learn from each failure.

The lessons I learned in failure?

  • Because I failed the first time, I can find success even in future failures;
  • Grant reviews are *grant* reviews, not judgements of your ability as a researcher, or the capability of your research team (regardless of Reviewer 3’s opinion);
  • Grant reviewers may be experts, they are not necessarily experts in the same field as your intended audience;
  • A single opinion is not a useful judgement of success or failure. In contrast, by looking at multiple opinions across reviews can provide valuable insight;
  • While each grant must display individual justifications for funding, similar comments across different grants can influence future writing;
  • Once you have finished what you control, move on to your next project. This maintains forward momentum in case of setbacks, allows you to set goals within your control, and apply the information learned to a new grant if the reviews come back before the next grant cycle; and REMEMBER
  • Although there is always a Reviewer 3, thank God for Reviewer 4!