Draft safety review: the rodent thyroid C-cell signal and its… — submissions
The 16 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
16 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The conclusion is stated more strongly than the certainty rating supports
The draft review of What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? was read against its registered protocol.
The draft rates the anchor outcome at low certainty and then states the finding in the conclusion without qualification. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
Point estimates are given without an interval
Having read the draft synthesis on What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?, the respondent puts one point to the committee.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
The review question is drawn too broadly to be answerable
This submission concerns the draft review of What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
Crossover trials are pooled with parallel-group trials without adjustment
The respondent has read What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? in draft and makes a single submission.
The analysis combines parallel-group and crossover designs by taking the first-period result from the crossovers, which discards half the data, or by taking the paired result, which is not comparable with an unpaired one. The methods do not say which was done.
The respondent proposes that the handling be stated and, where crossover data are used, that a sensitivity analysis excluding them be reported.
The secretariat accepts the requirement to state the handling and declines to require a sensitivity analysis in every case.
The methods section now states the handling of every non-parallel design. A sensitivity analysis is reported where crossover trials contribute more than a fifth of the weight.
Studies not published in English were excluded without assessment
The respondent notes that the review question — What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The respondent states that a language restriction was applied at screening and that for some of the compounds in scope a substantial literature is published in other languages.
The respondent proposes that the restriction be removed and the review re-run.
The secretariat accepts this submission in part. The restriction is removed prospectively and the records already excluded on language grounds have been retrieved and screened on title and abstract in translation. The full re-run proposed is not undertaken, and the limitation is recorded rather than concealed.
Language restriction is removed from the protocol for the series, the records previously excluded on that ground are listed with their screening outcome, and the residual limitation is stated in the abstract of this review.
References should carry a persistent identifier for every cited source
The respondent submits on What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?. The point would apply equally to any document in the series.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
Attrition is assessed in the risk-of-bias domains and not reported as a number
The respondent read the draft review of What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? and has confined this submission to a single matter.
The risk-of-bias assessment records attrition as a judgement. The included-studies table does not carry the proportion analysed against the proportion randomised, so a reader cannot see whether a judgement of low risk was made over five per cent attrition or twenty-five.
The respondent proposes a numeric attrition column in the included-studies table.
The respondent endorses the general approach taken in submission 001 and asks that it be extended to the matter identified here.
The secretariat accepts this submission. A domain judgement is more useful when the number behind it is visible.
The included-studies table now carries randomised, analysed and attrition columns, and the risk-of-bias judgement is made against the figure the reader can see.
Baseline imbalance in a small contributing trial is not remarked on
This is a submission on the draft review of What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?, made from a statistical standpoint.
The respondent identifies a small contributing trial with a baseline difference in the outcome variable, and states that the difference is large enough to account for a share of the reported effect.
The respondent asks that the trial be excluded or that the imbalance be addressed.
The respondent notes that submission 001 has already been made and confines this submission to a matter not covered by it.
The secretariat notes this submission. The imbalance is recorded in the risk-of-bias assessment for that trial, in the domain concerning the randomisation process, where the respondent may not have looked.
No amendment arises. The judgement and the supporting figures are already reported at domain level for that study, and a sensitivity analysis omitting it is reported in the results, which does not change the direction of the estimate.
The search strategy as published cannot be re-run
The respondent’s comment on the draft review of What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? arises from comparing the included set against the respondent’s own knowledge of the field.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
The respondent has read submission 001 above and makes this submission independently of it.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
Declared interests should appear on the document rather than on a separate page
The respondent submits on What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?, on a matter that is not specific to this draft but is visible in it.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
The axis of the forest plot exaggerates a small difference
The respondent has read the draft synthesis on What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? and makes one submission.
The plot is drawn on an axis spanning a narrow range, so an effect of no clinical consequence occupies most of the width of the figure. A reader who takes the visual impression rather than the number will overstate the finding.
The respondent proposes that a minimally important difference be marked on every forest plot where one has been established for the outcome.
This submission should be read alongside submission 008, which arises on the same draft.
The secretariat accepts the proposal where a minimally important difference exists and declines to invent one where it does not.
Forest plots now carry a marked minimally important difference wherever a published one exists for the outcome, with its source cited. Where none exists the figure says so beneath the axis.
Results at materially different follow-up durations are pooled
The respondent has read What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? and submits on a matter of presentation.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The respondent has read submission 009 with interest and adds one observation the secretariat may find useful.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
Patient-reported outcomes are collected by the trials and not reported by the review
Having read the draft under consultation, which concerns What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?, the respondent submits as follows.
The respondent states that several contributing trials measured quality of life and function and that the review reports neither, having selected outcomes on the basis of what could be pooled.
The respondent proposes that these outcomes be reported narratively where they cannot be pooled.
The secretariat accepts this submission in part. The outcomes are reported narratively. The proposal to pool them is declined because the instruments used are not comparable and pooling would produce a figure with no interpretation.
Patient-reported outcomes measured by any contributing trial are now reported narratively by instrument, with the instrument named and its minimum important difference stated where one is published, and the absence of a pooled estimate explained.
An indirect comparison is presented without an assessment of transitivity
The respondent submits on the draft synthesis addressing What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans?. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
Risk-of-bias judgements are reported as an overall rating without the domains
This submission concerns What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? and a convention used across the Institute’s output.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The respondent notes that submission 012 has already been made and confines this submission to a matter not covered by it.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
Two factual descriptions of the sponsor's programme are inaccurate
This submission addresses the draft synthesis on What is the evidence that glucagon-like peptide-1 receptor agonism increases the risk of medullary thyroid carcinoma in humans? from the standpoint of a reader who will use the summary and not the appendices.
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.