Draft synthesis: glucagon-like peptide-1 receptor agonists… — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Results at materially different follow-up durations are pooled
The respondent submits on In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on…. The point would apply equally to any document in the series.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
The axis of the forest plot exaggerates a small difference
The respondent has read the draft synthesis on In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… and makes one submission.
The plot is drawn on an axis spanning a narrow range, so an effect of no clinical consequence occupies most of the width of the figure. A reader who takes the visual impression rather than the number will overstate the finding.
The respondent proposes that a minimally important difference be marked on every forest plot where one has been established for the outcome.
The secretariat accepts the proposal where a minimally important difference exists and declines to invent one where it does not.
Forest plots now carry a marked minimally important difference wherever a published one exists for the outcome, with its source cited. Where none exists the figure says so beneath the axis.
An indirect comparison is presented without an assessment of transitivity
This submission addresses the draft synthesis on In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… from the standpoint of a reader who will use the summary and not the appendices.
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
Intention-to-treat and efficacy-estimand results are combined without distinction
This submission concerns the draft review of In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
The document is unreadable without specialist training
The respondent read the draft review of In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… and has confined this submission to a single matter.
The respondent states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
Absence of evidence is presented in a form a reader will take as negative evidence
The respondent has read In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… in draft and makes a single submission.
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
Regulatory assessment documents were not searched
The respondent read In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… in draft. The point applies to it and to the series generally.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
This point is adjacent to the one made in submission 002 and the respondent puts it in a form the secretariat can act on.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
A sponsor trial meeting the eligibility criteria was excluded
Having read the draft synthesis on In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on…, the respondent puts one point to the committee.
The submission is made on behalf of the sponsor. It identifies a completed trial of the sponsor's compound that meets the stated eligibility criteria and does not appear in the included set, and supplies the trial report and the registry record.
The sponsor asks that the trial be included and the estimate recomputed. No view is expressed on the direction the recomputation should take.
The respondent notes submission 003 above and does not repeat the ground it covers.
The secretariat accepts this submission in part. The trial does meet the criteria and has been included. The recomputed estimate is materially unchanged, which the response states explicitly so that the outcome of the correction is on the record.
The trial is added to the included set, the estimate and the certainty rating have been recomputed, and the screening record now states why the trial was missed, which was a database indexing gap rather than a screening judgement. The submission is identified as an industry submission.
The review protocol is described but its registration record is not linked
The respondent submits on the draft synthesis addressing In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
Overlapping primary studies across included reviews are counted more than once
The respondent’s comment on the draft review of In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… arises from comparing the included set against the respondent’s own knowledge of the field.
The draft is an overview of reviews and several included reviews share primary studies. The respondent states that the overview reports the total number of participants across reviews as though the sets were disjoint.
The respondent proposes that overlap be assessed and reported, and that no total be given without correcting for it.
The respondent’s submission overlaps with submission 006 and was prepared without sight of it.
The secretariat accepts this submission. An inflated participant total overstates the evidence base by an amount the reader cannot see.
Overlap between included reviews is now assessed at primary-study level and reported in a citation matrix, and participant totals are reported for the union of primary studies rather than as a sum across reviews.
Registered trials that never reported are not counted anywhere in the review
The draft review of In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… was read against its registered protocol.
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
The review question is drawn too broadly to be answerable
The respondent submits on In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on….
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
Submission 008 raises an adjacent matter. The respondent regards the two as separable and addresses only this one.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
The choice of effect measure is not justified and changes the appearance of the result
This is a submission on the draft review of In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on…, made from a statistical standpoint.
The draft reports relative effects for benefits and absolute effects for harms. The respondent states that the combination makes the finding look larger than the uniform presentation would, and that the choice should be justified or made uniform.
The respondent proposes that both relative and absolute effects be reported for every outcome.
The respondent supports submission 009 so far as it goes and adds the matter set out here.
The secretariat accepts this submission in part. Both measures are reported for every outcome where the baseline risk needed for the absolute effect can be stated. Where it cannot, the relative effect is reported alone with the reason.
Every outcome now reports the relative effect and, where an assumed baseline risk can be stated and sourced, the corresponding absolute effect, with the baseline risk and its source given in the same row.
Doses differing several-fold are pooled without examining dose-response
The respondent notes that the review question — In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
Declared interests should appear on the document rather than on a separate page
This submission concerns In adults with obesity and without type 2 diabetes, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on… and makes one point.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.