Peers Inside the Black Box: Deciding Excellence

Karen Mow

    Research output: Contribution to journalArticle

    Abstract

    Excellence is decided inside the grant selection process, in what some applicants call the ⿿black box�. This discussion examines the process from the perspective of peer reviewers as selection panel members and external assessors. It includes discussion of the ways that these peers review and rank applications for research funding. In particular, it seeks to reveal how peers define excellence and whether the ways that grant peer review is operationalised affects the ways that excellence is defined. Data from 23 semi-structured interviews with panel members and council staff are augmented by responses from 18 panel members and 70 external assessors collected in a survey of applicants. These responses form the basis of a discussion and reflection on how comparative selection decisions are made and the ways that external reviewers see the process of determining excellence. These data open to scrutiny the invisible hand of scientific opinion, the judgements scientists make within research council processes (Rip, 1993). Excellence emerged as a multi-faceted or polymorphic (Lamont, 2009) construction but clarity and writing dominated opinions about what was the best. Every panel member interviewed held that the difference between the best (5-10%) and the rest is the way that the Text component is written. Applications resonate, are compelling, convincing and shine off the page. They are like really good literature. Panel members believe that communicating the work of research is an integral part of the activity and that all researchers have an obligation to deliver clear stories about their work to the world at large, that in fact, as one influential Panel member said; get over it, it⿿s the job. In summary, the best applications are well articulated, professionally prepared and exciting to read. Panel members believe that people who can tell the story also do the best science.
    Original languageEnglish
    Pages (from-to)175-184
    Number of pages10
    JournalThe International Journal of Interdisciplinary Social Sciences
    Volume5
    Issue number10
    Publication statusPublished - 2011

    Fingerprint

    peer review
    applicant
    staff council
    grant
    obligation
    funding
    interview
    science
    literature

    Cite this

    @article{3c7f4a4db37f4a96b2c2173aed90ee95,
    title = "Peers Inside the Black Box: Deciding Excellence",
    abstract = "Excellence is decided inside the grant selection process, in what some applicants call the {\^a}¿¿black box{\^a}¿�. This discussion examines the process from the perspective of peer reviewers as selection panel members and external assessors. It includes discussion of the ways that these peers review and rank applications for research funding. In particular, it seeks to reveal how peers define excellence and whether the ways that grant peer review is operationalised affects the ways that excellence is defined. Data from 23 semi-structured interviews with panel members and council staff are augmented by responses from 18 panel members and 70 external assessors collected in a survey of applicants. These responses form the basis of a discussion and reflection on how comparative selection decisions are made and the ways that external reviewers see the process of determining excellence. These data open to scrutiny the invisible hand of scientific opinion, the judgements scientists make within research council processes (Rip, 1993). Excellence emerged as a multi-faceted or polymorphic (Lamont, 2009) construction but clarity and writing dominated opinions about what was the best. Every panel member interviewed held that the difference between the best (5-10{\%}) and the rest is the way that the Text component is written. Applications resonate, are compelling, convincing and shine off the page. They are like really good literature. Panel members believe that communicating the work of research is an integral part of the activity and that all researchers have an obligation to deliver clear stories about their work to the world at large, that in fact, as one influential Panel member said; get over it, it{\^a}¿¿s the job. In summary, the best applications are well articulated, professionally prepared and exciting to read. Panel members believe that people who can tell the story also do the best science.",
    keywords = "Peer Review, Research Grant Funding, Grant Selection, Excellence",
    author = "Karen Mow",
    year = "2011",
    language = "English",
    volume = "5",
    pages = "175--184",
    journal = "The International Journal of Interdisciplinary Social Sciences",
    issn = "1833-1882",
    publisher = "Common Ground Publishing",
    number = "10",

    }

    Peers Inside the Black Box: Deciding Excellence. / Mow, Karen.

    In: The International Journal of Interdisciplinary Social Sciences, Vol. 5, No. 10, 2011, p. 175-184.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Peers Inside the Black Box: Deciding Excellence

    AU - Mow, Karen

    PY - 2011

    Y1 - 2011

    N2 - Excellence is decided inside the grant selection process, in what some applicants call the ⿿black box�. This discussion examines the process from the perspective of peer reviewers as selection panel members and external assessors. It includes discussion of the ways that these peers review and rank applications for research funding. In particular, it seeks to reveal how peers define excellence and whether the ways that grant peer review is operationalised affects the ways that excellence is defined. Data from 23 semi-structured interviews with panel members and council staff are augmented by responses from 18 panel members and 70 external assessors collected in a survey of applicants. These responses form the basis of a discussion and reflection on how comparative selection decisions are made and the ways that external reviewers see the process of determining excellence. These data open to scrutiny the invisible hand of scientific opinion, the judgements scientists make within research council processes (Rip, 1993). Excellence emerged as a multi-faceted or polymorphic (Lamont, 2009) construction but clarity and writing dominated opinions about what was the best. Every panel member interviewed held that the difference between the best (5-10%) and the rest is the way that the Text component is written. Applications resonate, are compelling, convincing and shine off the page. They are like really good literature. Panel members believe that communicating the work of research is an integral part of the activity and that all researchers have an obligation to deliver clear stories about their work to the world at large, that in fact, as one influential Panel member said; get over it, it⿿s the job. In summary, the best applications are well articulated, professionally prepared and exciting to read. Panel members believe that people who can tell the story also do the best science.

    AB - Excellence is decided inside the grant selection process, in what some applicants call the ⿿black box�. This discussion examines the process from the perspective of peer reviewers as selection panel members and external assessors. It includes discussion of the ways that these peers review and rank applications for research funding. In particular, it seeks to reveal how peers define excellence and whether the ways that grant peer review is operationalised affects the ways that excellence is defined. Data from 23 semi-structured interviews with panel members and council staff are augmented by responses from 18 panel members and 70 external assessors collected in a survey of applicants. These responses form the basis of a discussion and reflection on how comparative selection decisions are made and the ways that external reviewers see the process of determining excellence. These data open to scrutiny the invisible hand of scientific opinion, the judgements scientists make within research council processes (Rip, 1993). Excellence emerged as a multi-faceted or polymorphic (Lamont, 2009) construction but clarity and writing dominated opinions about what was the best. Every panel member interviewed held that the difference between the best (5-10%) and the rest is the way that the Text component is written. Applications resonate, are compelling, convincing and shine off the page. They are like really good literature. Panel members believe that communicating the work of research is an integral part of the activity and that all researchers have an obligation to deliver clear stories about their work to the world at large, that in fact, as one influential Panel member said; get over it, it⿿s the job. In summary, the best applications are well articulated, professionally prepared and exciting to read. Panel members believe that people who can tell the story also do the best science.

    KW - Peer Review

    KW - Research Grant Funding

    KW - Grant Selection

    KW - Excellence

    M3 - Article

    VL - 5

    SP - 175

    EP - 184

    JO - The International Journal of Interdisciplinary Social Sciences

    JF - The International Journal of Interdisciplinary Social Sciences

    SN - 1833-1882

    IS - 10

    ER -