Draft synthesis: Cardiovascular outcomes with glucagon-like… — submissions
The 13 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
13 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Regulatory assessment documents were not searched
The respondent notes that the review question — In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
Results at materially different follow-up durations are pooled
This submission addresses the draft synthesis on In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… from the standpoint of a reader who will use the summary and not the appendices.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The respondent has read submission 001 and asks that this submission be considered with it.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
Point estimates are given without an interval
This submission concerns the draft review of In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Studies not published in English were excluded without assessment
The respondent submits on the draft synthesis addressing In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent states that a language restriction was applied at screening and that for some of the compounds in scope a substantial literature is published in other languages.
The respondent proposes that the restriction be removed and the review re-run.
The secretariat accepts this submission in part. The restriction is removed prospectively and the records already excluded on language grounds have been retrieved and screened on title and abstract in translation. The full re-run proposed is not undertaken, and the limitation is recorded rather than concealed.
Language restriction is removed from the protocol for the series, the records previously excluded on that ground are listed with their screening outcome, and the residual limitation is stated in the abstract of this review.
The document should state what a reader ought to do
The draft review of In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… was read against its registered protocol.
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The respondent’s submission overlaps with submission 001 and was prepared without sight of it.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
A surrogate outcome is used as the anchor without validation evidence
The respondent submits on In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse….
The anchor outcome in the draft is a surrogate. The respondent states that the relationship between the surrogate and the outcome a decision turns on is itself an evidential question, and that the draft assumes it.
The respondent proposes that no surrogate serve as an anchor.
The secretariat accepts this submission in part. The anchor is retained where the surrogate is the only outcome the contributing trials measured, and the validation question is addressed rather than assumed.
Where the anchor is a surrogate, the synthesis now states the evidence for the surrogate relationship, rates it separately, and downgrades the anchor rating for indirectness accordingly rather than carrying the surrogate as though it were the outcome of interest.
Declared interests should appear on the document rather than on a separate page
The respondent submits on In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse…, on a matter that is not specific to this draft but is visible in it.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
Studies using different outcome definitions are pooled into one estimate
This is a submission on the draft review of In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse…, made from a statistical standpoint.
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
Efficacy outcomes are rated for certainty and harms are not
The respondent has read the draft synthesis on In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… and makes one submission.
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
Analytical surveys are treated as evidence about suppliers when they establish only what was in a sample
The respondent read In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… in draft. The point applies to it and to the series generally.
Several included surveys purchased material anonymously and analysed it. The respondent states that where the chain from a named supplier to the analysed vial cannot be documented, the result describes a sample and not a supplier.
The respondent proposes that provenance be an explicit eligibility dimension, with surveys reported separately according to whether it could be established.
The secretariat accepts this submission. Moderate certainty evidence from the included surveys supports statements about material circulating in a market; it does not support statements about any named supplier's output.
Provenance is now assessed for every included survey and reported as a study characteristic, surveys without establishable provenance are reported separately rather than pooled with those that have it, and no supplier-level inference is drawn from a sample-level result.
Doses differing several-fold are pooled without examining dose-response
The respondent read the draft review of In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… and has confined this submission to a single matter.
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
Statistical heterogeneity is treated as though it measured clinical heterogeneity
The respondent’s comment on the draft review of In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse… arises from comparing the included set against the respondent’s own knowledge of the field.
The draft reports a heterogeneity statistic and proceeds to pool where it is low. The respondent states that a low statistic in a small set of trials is uninformative, and that clinical and methodological similarity should be assessed before any statistic is consulted.
The respondent proposes that the decision to pool be justified on clinical grounds first and that the statistic be reported as a description rather than used as a threshold.
The secretariat accepts this submission in part. The decision to pool is now made on clinical and methodological grounds and stated as such. The statistic continues to be reported, because readers expect it and its absence would be read as concealment.
The decision to pool is now justified in prose against the population, intervention, comparator and outcome before any statistic is presented, and the heterogeneity statistic is reported with its confidence interval and a note of the number of contributing studies.
The timepoint at which each outcome was extracted was chosen after the data were seen
Having read the draft synthesis on In adults at elevated cardiovascular risk, what is the effect of a glucagon-like peptide-1 receptor agonist compared with placebo on major adverse…, the respondent puts one point to the committee.
The methods state that outcomes were extracted at the latest available timepoint. Several contributing trials report the same outcome at three timepoints, and taking the latest is a choice that can be made in the knowledge of what each shows.
The respondent proposes that the extraction timepoint be prespecified in the protocol, and that where it was not, the review report the estimate at every reported timepoint so that the choice is visible.
The secretariat accepts this submission. An extraction rule applied after the data are visible is a degree of freedom and should be recorded as one.
The extraction timepoint is now prespecified in the protocol for new reviews. Where a review predates the requirement, estimates are reported at every timepoint the contributing trials report.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.