Draft policy on the register of members and reviewers — submissions
The 13 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
13 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Independence is asserted and the mechanism securing it is not described
The respondent has read policy on the register of members and reviewers in draft and makes one submission.
The document states that sponsorship does not influence assessment. It does not describe what prevents it from doing so, and an assertion without a mechanism is not verifiable.
The respondent proposes that the separation between the party accepting funding and the party making assessments be described in operational terms.
The secretariat accepts this submission.
The governance document now describes the separation in operational terms: who accepts funding, who makes assessments, that the two sets do not overlap, and that assessment staff are not informed of placement decisions before publication.
Assessment committee meetings are held in private
The respondent submits on policy on the register of members and reviewers. Governance documents are read by very few people and relied on by everyone who reads anything else the Institute publishes.
Comparable bodies admit observers to at least part of a meeting. The document does not say whether the Institute has considered doing so.
The respondent proposes that part of each meeting be open, or that the reason for holding meetings in private be stated.
The secretariat notes this submission and states the position rather than changing it.
Meetings are held in private and the reason is now stated: the committee assesses draft ratings that will change before publication, and an observed deliberation on a draft rating tends to produce a published deliberation rather than a candid one. The minutes and the dissenting positions are published.
What the site collects about a reader is not stated in terms a reader can check
The respondent notes that policy on the register of members and reviewers governs situations that will arise rarely and matter greatly when they do, and submits accordingly.
The respondent states that a body assessing evidence should be checkable in its own conduct, and that a general privacy statement does not permit a reader to verify what is collected.
The respondent proposes that the statement enumerate what is collected and that it be verifiable from the page itself.
The respondent has read submission 002 and asks that this submission be considered with it.
The secretariat accepts this submission. An unverifiable statement about data is exactly the kind of claim the Institute declines to accept from others.
The statement now enumerates what is collected and what is not, and is written so that a reader can verify it from the page source, which is the only form of the statement that can be checked without trusting the Institute.
Recusal should follow automatically from any declared financial interest
Having read the draft of policy on the register of members and reviewers, the respondent puts one point to the secretariat.
The respondent states that case-by-case recusal decisions require the body making them to judge its own members, and that an automatic rule removes the judgement and with it the appearance of partiality.
The respondent proposes automatic recusal on any declared financial interest in a named company.
The respondent has read submission 003 and puts a further matter to the secretariat.
The secretariat does not accept this submission. An automatic rule of that breadth would recuse most of the people competent to assess the evidence, and the Institute would be left with assessors chosen for their distance from the field.
Recusal remains determined against the specific document, with the decision, its reasons and the person recused published in the document front matter. The submission is published in full because it identifies a genuine tension the Institute has resolved in one direction rather than removed.
The search date is not on the face of the document
The draft of policy on the register of members and reviewers was read for what it obliges the Institute to do rather than for what it says the Institute intends.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
This submission should be read alongside submission 004, which arises on the same draft.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
The Institute publishes no telephone number and no messaging channel
The respondent’s comment on policy on the register of members and reviewers is made from the standpoint of a person who might be affected by a decision taken under it.
The respondent states that a body inviting submissions ordinarily provides a channel for questions about the process, and that the written address alone is a high barrier for a first-time respondent.
The respondent asks that a route for procedural questions be provided.
The secretariat notes this submission. The written route is the only route and that is a deliberate constitutional position rather than an omission.
No amendment arises. The Institute holds no telephone number and uses no messaging channel, because a submission made in writing is a document that can be published with its response and an oral exchange is not. Guidance on making a submission has been expanded so that the written route requires no prior enquiry to use.
No period is stated within which a submission will be answered
The respondent has read policy on the register of members and reviewers and submits on a matter of presentation.
A respondent who has made a submission has no basis for knowing whether silence means the submission is being considered or has been lost.
The respondent proposes a stated period, with an acknowledgement on receipt and a substantive response within it.
The secretariat accepts this submission.
Receipt is acknowledged within five working days and a substantive response is published within eight weeks of the window closing. Where that period cannot be met the reason is published in its place.
Declared interests should appear on the document rather than on a separate page
The respondent read policy on the register of members and reviewers in draft and has confined this submission to one clause.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The respondent’s submission overlaps with submission 006 and was prepared without sight of it.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
The position of a contributor who withdraws consent is not stated
The respondent submits on policy on the register of members and reviewers, on a matter that is not specific to this draft but is visible in it.
The respondent states that consent to be named is obtained at submission and that the policy is silent on what happens if it is later withdrawn, which leaves both the Institute and the contributor without a rule.
The respondent proposes that the position be stated in advance and be the same for everyone.
The secretariat accepts this submission. A rule made at the point of the first withdrawal is a rule made about a particular person.
The policy now states that a contributor may withdraw from the register at any time and that the register entry is removed, while a signed submission already published remains published, because the consultation record is a record of what was said and by whom. The position is stated at the point consent is given.
Whether external reviewers are paid is not stated
This submission concerns the draft of policy on the register of members and reviewers. The respondent has served on committees operating comparable regimes and the observation draws on that.
The respondent states that payment or its absence bears on who is able to review and on what the Institute may expect of them, and that the policy does not address it.
The respondent proposes that the position be stated and that any payment be disclosed at the level of the individual document.
The respondent notes submission 009 above and does not repeat the ground it covers.
The secretariat accepts this submission. Payment is an interest and the policy treats interests as facts to be published.
The policy now states whether external review is remunerated, records any payment made in respect of a document in that document's front matter, and treats a payment received from the Institute as a declarable interest in the same way as any other.
The secretariat answers submissions and is not itself named
This is a submission on policy on the register of members and reviewers, from a respondent representing readers rather than practitioners.
The respondent states that responses are attributed to the secretariat as a body, and that a reader cannot tell who wrote a response or whether that person had a relevant interest.
The respondent proposes that responses be signed individually.
The secretariat accepts this submission in part. Responses are attributed to the named members of the secretariat responsible for the document rather than to individuals per response, and the interests of those members are published.
Every comment period now names the secretariat members responsible for it, with their declared interests, in the consultation front matter, so that the responses can be read against the interests of the people who wrote them.
A party that is assessed and also sponsors should be disclosed on its own assessment
The respondent read policy on the register of members and reviewers in draft. The point applies to it and to the series generally.
Disclosure on the funding page is necessary and is not sufficient. The reader who most needs to know is the one reading that party’s assessment, and that reader has no reason to visit the funding page.
The respondent proposes that the disclosure appear on the assessment itself, in the same visual weight as the assessment summary.
This submission should be read alongside submission 009, which arises on the same draft.
The secretariat accepts this submission without qualification. It was a real defect.
The disclosure now appears as a numbered section on the dossier of any assessed party that sponsors the site, as well as on the funding page.
A submission the Institute declines to publish leaves no trace
This submission addresses policy on the register of members and reviewers from the standpoint of an outside body that would have to rely on it.
The policy permits a submission to be declined where it is abusive or names an individual. Nothing records that a submission was received and declined, so the published set cannot be distinguished from the received set.
The respondent proposes a count of declined submissions with the ground for each.
The secretariat notes this submission and has adopted part of it.
The number of submissions declined and the ground for each is now published on the disposition page of the consultation concerned. The submissions themselves are not published, for the reasons that caused them to be declined.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.