Draft synthesis: Counter-ion content and its effect on… — submissions
The 16 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
16 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The document is unreadable without specialist training
The respondent submits on the draft review of How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?. This submission concerns how the contributing results were obtained rather than what they show.
The respondent states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
The document should not describe uses outside the approved indication
The respondent has read the draft synthesis on How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? and makes one submission about the analytical evidence it pools.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
References should carry a persistent identifier for every cited source
The respondent has read How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? and submits on a matter of presentation.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
Risk-of-bias judgements are reported as an overall rating without the domains
Having read the draft under consultation, which concerns How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?, the respondent submits as follows.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
Submission 001 concerns the same document. The respondent’s point is a different one.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
Whether the analyst knew the source of each sample is not reported
The respondent’s comment on the draft review of How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? arises from comparing the pooled figures against the reports they were taken from.
The contributing surveys do not state whether the laboratory knew which source each sample came from.
The respondent states that integration decisions and the treatment of borderline results involve judgement, and that knowledge of the source is capable of influencing that judgement without any intention that it should.
The respondent proposes that blinding be recorded where it is reported and requested from contributing authors where it is not.
The respondent read submission 004 after drafting this one and has not altered it, the two points being distinct.
The secretariat notes this submission and has recorded blinding where the contributing authors reported it, which is a minority of the set.
No amendment to the estimates arises. Blinding is now recorded survey by survey, the absence of the information is stated rather than assumed either way, and the point is carried into the limitations.
Studies using different outcome definitions are pooled into one estimate
Having read the draft synthesis on How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?, the respondent puts one point about the underlying measurements to the committee.
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
The respondent supports submission 004 so far as it goes and adds the matter set out here.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
The chain of custody between purchase and analysis is not described
This submission concerns the draft synthesis on How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?. The respondent works with determinations of this kind and the observation arises from that.
The contributing surveys describe what was analysed and do not describe how the material travelled between the seller and the instrument.
The respondent states that storage temperature and time in transit affect several of the attributes reported, so a difference between surveys may be a difference in handling rather than in supply.
The respondent proposes that transit and storage conditions be recorded for every contributing survey.
The secretariat notes this submission and records that it asked each contributing author for the information the respondent identifies.
No amendment beyond disclosure arises, because the information is not held for most of the contributing surveys. The synthesis now states, survey by survey, whether handling conditions were reported, and the limitation is carried into the certainty rating rather than left in the discussion.
Contributing surveys anonymise their sources to different degrees
This submission addresses the draft synthesis on How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? from the standpoint of a reader who will act on the summary figure.
Some contributing surveys name the sources they bought from, some describe them by category, and some do not describe them at all.
The respondent states that a synthesis drawing on all three cannot report by source, and that a reader may take the category descriptions as comparable when they were not defined in common.
The respondent proposes that the level of source description be recorded for each survey and that no analysis be reported by source.
The secretariat notes this submission and records that the practice the respondent describes is a property of the contributing literature rather than of the draft.
No analysis by source is reported. The level of source description is now recorded survey by survey, and the review states that it draws no conclusion about any individual source and that its findings are properties of the sampled literature.
Regulatory assessment documents were not searched
The respondent submits on How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?. The point would apply equally to any document in the series.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
A superseded version should remain reachable from the version that replaced it
The respondent submits on How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?, on a matter that is not specific to this draft but is visible in it.
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The respondent endorses the general approach taken in submission 002 and asks that it be extended to the matter identified here.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
A funnel plot is presented for a set too small to interpret it
The respondent notes that the review question — How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? — is answerable only if the contributing determinations measured the same quantity, and submits with that in view.
The draft includes a funnel plot for an outcome contributed by fewer than ten studies. The respondent states that asymmetry cannot be assessed reliably at that number and that presenting the plot invites a conclusion the data cannot support.
The respondent proposes that the plot be removed and replaced with a statement that publication bias could not be assessed.
The secretariat accepts this submission. Presenting an uninterpretable graphic is not a neutral act.
The funnel plot is removed for every outcome contributed by fewer than ten studies, and replaced by a statement that small-study effects could not be assessed at that number, together with the count of registered trials identified without posted results.
The search strategy as published cannot be re-run
The respondent read the draft review of How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? and has confined this submission to a single matter of measurement.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
The sampling frame of each contributing survey is not reported
The respondent read How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? in draft. The point applies to it and to the series generally.
The included surveys each obtained material from a set of sources, and the draft reports what was found without reporting how the sources were chosen.
The respondent states that a survey that bought from the most visible sellers and a survey that bought at random from a directory answer different questions, and that pooling them without recording the difference produces a figure that describes neither.
The respondent proposes that the sampling frame, the number of sources approached and the number that supplied be tabulated for every contributing survey.
The respondent has read submission 003 above and makes this submission independently of it.
The secretariat accepts this submission. How material was obtained determines what a survey can support, and the draft treated the surveys as interchangeable.
Every contributing survey now carries a sampling row stating the frame, the number of sources approached and the number supplying, and the synthesis reports separately for surveys whose frames are not comparable.
The conclusion is stated more strongly than the certainty rating supports
The respondent has read How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? in draft and makes a single submission.
The draft rates the anchor outcome at low certainty and then states the finding in the conclusion without qualification. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
Two included studies do not meet the registered eligibility criteria
The draft review of How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported? was read against the certificates and reports it draws on.
The respondent identifies two studies in the included set whose duration falls below the minimum stated in the protocol, and one excluded study that appears to meet every criterion.
The respondent proposes that the three be reassessed and that the outcome of the reassessment be recorded whichever way it goes.
The secretariat accepts this submission in part. On reassessment, one of the two included studies does fall below the minimum duration and has been removed. The second reports a duration that met the criterion in the protocol version registered at the time. The excluded study was excluded for a reason not clearly recorded, which was a documentation failure.
One study has been removed from the included set and the estimate recomputed, the exclusion reason for the third study has been corrected in the excluded-studies table, and the screening decisions are now recorded against the protocol version in force at the time of screening.
Results obtained by different methods are pooled without recording the method
This is a submission on the draft review of How large is the discrepancy between labelled peptide mass and delivered peptide mass when counter-ion and water content are not reported?, made from the standpoint of a laboratory that would have to reproduce the determinations.
The contributing results were obtained by more than one determination, and the draft reports a single distribution.
The respondent states that the determinations do not measure the same quantity, that the difference between them is systematic rather than random, and that a pooled distribution across methods has no referent.
The respondent proposes that every result carry the method that produced it and that pooling occur only within a method.
This point is adjacent to the one made in submission 008 and the respondent puts it in a form the secretariat can act on.
The secretariat accepts this submission. A figure whose method is not recorded cannot be compared with another figure, and the draft pooled across the difference.
Every contributing result now carries its determination, pooling occurs only within a determination, and where a survey did not report the method the result is listed as method not stated and excluded from the pooled estimates.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.