Draft certainty index — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
References should carry a persistent identifier for every cited source
The respondent notes that certainty index will govern a great many documents, and submits accordingly.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
The same quantity is presented differently in two series
This is a submission on certainty index, from a respondent who reads these documents on paper.
The respondent states that a certainty rating is displayed one way in the monograph series and another in the synthesis series, and that a reader moving between them cannot be sure the two mean the same thing.
The respondent proposes a single presentation across all series.
The respondent’s submission overlaps with submission 001 and was prepared without sight of it.
The secretariat accepts this submission. A difference in presentation is read as a difference in meaning.
Certainty ratings are now presented identically across every series, using the same badge, the same wording and the same accompanying formulation, and the presentation is specified in the series structure document rather than left to each series.
The anchor outcome for an indication is not the outcome a decision turns on
Having read the draft of certainty index, the respondent puts one point to the secretariat.
The respondent, a practising clinician, states that for two indications in the draft set the anchor is the outcome most often measured rather than the outcome that would determine whether to treat.
The respondent proposes that the anchors be changed and gives the alternatives.
This submission should be read alongside submission 002, which arises on the same draft.
The secretariat accepts this submission in part. One anchor is changed. The other is retained because no contributing trial measures the alternative, and an anchor no study reports produces a series of documents that can report nothing.
One anchor outcome is changed to the outcome proposed, and the second is retained with the alternative recorded as the preferred anchor and listed as an evidence gap, so that the reason for the compromise is on the record.
A document is split across parts at a point that separates a claim from its support
The respondent submits on certainty index. A document series is a set of promises about where things will be found, and this submission concerns one of them.
The evidence is on one page and the certainty assessment that qualifies it on another. A reader who lands on the first from a search engine gets the claim without the qualification.
The respondent proposes that every part carry the certainty rating for the material it presents.
The secretariat accepts the proposal and applies it in the metadata rather than by restructuring.
Every part now carries the certainty rating in its own metadata header and repeats it beside any estimate it reproduces, so that no page presents an estimate without its rating.
Indications for which compounds are supplied are absent from the set
The respondent has read certainty index in draft and makes a single submission.
The respondent states that the indication set is drawn from approved and investigated uses, and that several uses for which compounds are widely supplied have no entry, so no document can address them.
The respondent proposes that such uses be added as indications with no anchor outcome and an explicit statement of the evidence position.
The respondent has read submission 002 and asks that this submission be considered with it.
The secretariat accepts this submission. An indication set that omits the uses a reader is most likely to encounter cannot serve that reader.
Uses observed in supply are now included in the indication set, marked as having no approved status and no anchor outcome, and each carries an explicit statement that no trial evidence was identified where that is the case.
An identifier that encodes the document type will have to change when the type does
This submission concerns the draft of certainty index. The respondent’s interest is in retrieval: whether a document can be found again, and cited, after it has been superseded.
The respondent states that the proposed identifier embeds the series and the type, and that a document reclassified after publication would then require a new identifier, breaking every citation to it.
The respondent proposes that the identifier carry no semantic content and that type be a field rather than part of the identifier.
The secretariat accepts this submission. An identifier that has to change is not an identifier.
The identifier is now opaque and permanent from first publication, the series and type are recorded as fields that may change, and a reclassified document keeps the identifier it was issued with.
The identifier encodes the document type, which is a property that can change
This submission concerns certainty index and makes one point.
An identifier containing a type segment will have to change if a document moves between series, and an identifier that changes is not an identifier.
The respondent proposes that the type segment be dropped or that a commitment be published never to change one.
The secretariat accepts the second limb.
The identifier is retained in its present form and a standing commitment is published that no identifier will be reassigned. A document that moves between series keeps the identifier it was issued with, and the type segment is documented as historical rather than descriptive.
Absence of evidence is presented in a form a reader will take as negative evidence
The respondent has read certainty index in draft and makes one submission.
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
A withdrawn document should not simply disappear
The respondent read certainty index in draft and has confined this submission to a single provision.
The respondent states that a reader arriving from an external citation at a withdrawn document needs to know that it existed, that it was withdrawn and why, and that a redirect to an index gives none of that.
The respondent proposes a notice at the original address.
The secretariat accepts this submission. A silent removal makes the citation that led there unresolvable.
A withdrawn document is now replaced at its own address by a notice recording its identifier, its title, the date and ground of withdrawal, and a link to whatever supersedes it, and the withdrawal is recorded in the corrections log.
The front matter delays the reader from reaching the assessment
The respondent read certainty index in draft. The point applies to it and to the series generally.
The respondent states that a reader must pass through identifiers, dates, contributor lists and declarations before reaching any content, and that the most important sentence in the document is the furthest from the top.
The respondent proposes that the front matter be moved to the end.
The secretariat accepts this submission in part. The assessment statement is moved to the top. The front matter is not moved to the end, because it is what allows the assessment to be checked and a document whose provenance is at the bottom will be quoted without it.
Every document now opens with its assessment statement and its certainty rating, with the identifiers, dates and declarations following immediately beneath rather than above, so that the first thing read is the finding and the second is its provenance.
A long index is paginated in a way that makes an item hard to find again
The draft of certainty index was read from the standpoint of someone who will navigate the series rather than write for it.
An item on page eleven of an index moves as documents are added, so a reference to it does not hold.
The respondent proposes stable per-item addresses in preference to page references.
The secretariat notes this submission. The practice it asks for is already the practice.
Every item in every index has a stable address of its own and pagination is a means of reaching it, never an address. Nothing in the series should be cited by page number and the citation block never offers one.
A section with nothing in it is emitted with a heading and a sentence saying so
This submission addresses certainty index from the standpoint of a reader arriving from a search engine rather than from the front page.
The fixed structure produces sections carrying a heading and one sentence recording that there is nothing to report. Repeated across a series this is a substantial amount of text saying nothing.
The respondent proposes that an empty section be folded into its parent with a single line, rather than given a heading of its own.
The secretariat accepts this submission.
A section with no content is now folded into its parent as a single recorded line. The fact of the absence is preserved, because an absent section and an absent statement are different, and the heading is not.
A document does not show what draws on it
The respondent submits on certainty index. The point would apply equally to any document in the series.
A trial abstract is cited by monographs and syntheses. The abstract itself does not list them, so a reader who finds the trial cannot see how it has been used.
The respondent proposes a citing-documents list on every document that others draw on.
The secretariat accepts this submission.
Every trial abstract and every bibliography entry now carries a list of the documents that draw on it, generated from the citations themselves so that it cannot fall out of step.
The series scope is drawn widely enough to absorb documents that belong elsewhere
The respondent submits on certainty index.
The respondent states that the scope as drafted would admit documents that are neither assessments nor standards, and that a series which admits everything ceases to tell a reader anything about what a document is.
The respondent proposes a narrower scope with an exclusion list.
The secretariat accepts this submission in part. An exclusion list is added. The scope itself is not narrowed further, because the excluded classes are better handled by naming them than by tightening a boundary that would then be argued over.
The series scope now carries an explicit exclusion list naming the document classes that do not belong to it and stating where each belongs instead, so that a placement decision is checkable rather than editorial.
Point estimates are given without an interval
The respondent’s comment on certainty index arises from consulting the series for reference rather than writing for it.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The respondent has read submission 001 and puts a further matter to the secretariat.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.