Draft safety review: delayed gastric emptying and procedural… — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
A surrogate outcome is used as the anchor without validation evidence
The respondent has read the draft synthesis on Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? and makes one submission.
The anchor outcome in the draft is a surrogate. The respondent states that the relationship between the surrogate and the outcome a decision turns on is itself an evidential question, and that the draft assumes it.
The respondent proposes that no surrogate serve as an anchor.
The secretariat accepts this submission in part. The anchor is retained where the surrogate is the only outcome the contributing trials measured, and the validation question is addressed rather than assumed.
Where the anchor is a surrogate, the synthesis now states the evidence for the surrogate relationship, rates it separately, and downgrades the anchor rating for indirectness accordingly rather than carrying the surrogate as though it were the outcome of interest.
Attrition is assessed in the risk-of-bias domains and not reported as a number
The respondent read the draft review of Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? and has confined this submission to a single matter.
The risk-of-bias assessment records attrition as a judgement. The included-studies table does not carry the proportion analysed against the proportion randomised, so a reader cannot see whether a judgement of low risk was made over five per cent attrition or twenty-five.
The respondent proposes a numeric attrition column in the included-studies table.
Submission 001 raises an adjacent matter. The respondent regards the two as separable and addresses only this one.
The secretariat accepts this submission. A domain judgement is more useful when the number behind it is visible.
The included-studies table now carries randomised, analysed and attrition columns, and the risk-of-bias judgement is made against the figure the reader can see.
The document is unreadable without specialist training
The draft review of Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? was read against its registered protocol.
The respondent states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
The review protocol is described but its registration record is not linked
The respondent submits on the draft synthesis addressing Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
Crossover trials are pooled with parallel-group trials without adjustment
The respondent has read Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? in draft and makes a single submission.
The analysis combines parallel-group and crossover designs by taking the first-period result from the crossovers, which discards half the data, or by taking the paired result, which is not comparable with an unpaired one. The methods do not say which was done.
The respondent proposes that the handling be stated and, where crossover data are used, that a sensitivity analysis excluding them be reported.
The secretariat accepts the requirement to state the handling and declines to require a sensitivity analysis in every case.
The methods section now states the handling of every non-parallel design. A sensitivity analysis is reported where crossover trials contribute more than a fifth of the weight.
The same concept is given three different names in one document
This submission addresses the draft synthesis on Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? from the standpoint of a reader who will use the summary and not the appendices.
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
Studies not published in English were excluded without assessment
Having read the draft under consultation, which concerns Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?, the respondent submits as follows.
The respondent states that a language restriction was applied at screening and that for some of the compounds in scope a substantial literature is published in other languages.
The respondent proposes that the restriction be removed and the review re-run.
The respondent notes submission 001 above and does not repeat the ground it covers.
The secretariat accepts this submission in part. The restriction is removed prospectively and the records already excluded on language grounds have been retrieved and screened on title and abstract in translation. The full re-run proposed is not undertaken, and the limitation is recorded rather than concealed.
Language restriction is removed from the protocol for the series, the records previously excluded on that ground are listed with their screening outcome, and the residual limitation is stated in the abstract of this review.
A sponsor trial meeting the eligibility criteria was excluded
The respondent submits on Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?.
The submission is made on behalf of the sponsor. It identifies a completed trial of the sponsor's compound that meets the stated eligibility criteria and does not appear in the included set, and supplies the trial report and the registry record.
The sponsor asks that the trial be included and the estimate recomputed. No view is expressed on the direction the recomputation should take.
The secretariat accepts this submission in part. The trial does meet the criteria and has been included. The recomputed estimate is materially unchanged, which the response states explicitly so that the outcome of the correction is on the record.
The trial is added to the included set, the estimate and the certainty rating have been recomputed, and the screening record now states why the trial was missed, which was a database indexing gap rather than a screening judgement. The submission is identified as an industry submission.
The conclusion is stated more strongly than the certainty rating supports
The respondent’s comment on the draft review of Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? arises from comparing the included set against the respondent’s own knowledge of the field.
The draft rates the anchor outcome at low certainty and then states the finding in the conclusion without qualification. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
The search was limited by publication date and the limit is not justified
This submission concerns the draft review of Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The search strategy carries a start date some years before the review. No reason is given. If the limit reflects the date of first synthesis of the compound it is defensible and should be said; if it reflects convenience it is a source of bias.
The respondent proposes that every date limit carry its justification in the search-strategy section.
The secretariat accepts the requirement to justify and declines to remove limits that are justified.
Every date limit in the series now carries its justification in the search-strategy section. Limits that could not be justified on re-examination have been removed and the affected searches re-run.
Results at materially different follow-up durations are pooled
This is a submission on Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The respondent read submission 010 after drafting this one and has not altered it, the two points being distinct.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
A single-trial result is presented in the visual form of a pooled estimate
This is a submission on the draft review of Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?, made from a statistical standpoint.
An outcome contributed by one trial is presented in the same forest plot format as outcomes contributed by several. The respondent states that the format carries an implication of replication that a single trial does not have.
The respondent proposes that single-trial outcomes be presented differently and labelled as unreplicated.
The respondent supports submission 006 so far as it goes and adds the matter set out here.
The secretariat accepts this submission. The form of a graphic is part of what it asserts.
Outcomes contributed by a single trial are now reported without a pooled diamond, labelled as unreplicated, and downgraded for imprecision or inconsistency according to the framework rather than presented as a synthesis.
The comparator was given at a dose below the one in general use
The respondent notes that the review question — Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? — is answerable only if the contributing trials measured the same thing, and submits with that in view.
Two contributing trials compare the intervention against an active comparator titrated to a dose lower than that reached in ordinary practice. Pooling them with trials using a full comparator dose produces an estimate that flatters the intervention.
The respondent proposes that comparator dose be tabulated in the included-studies table and that a sensitivity analysis restricted to full-dose comparators be reported.
The secretariat accepts this submission. Comparator dose is a condition of an effect estimate and was not being recorded.
Comparator dose is now a column in the included-studies table, and a sensitivity analysis restricted to comparators at the dose in general use is reported wherever the trials permit it.
A funnel plot is presented for a set too small to interpret it
This submission concerns Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation? and makes one point.
The draft includes a funnel plot for an outcome contributed by fewer than ten studies. The respondent states that asymmetry cannot be assessed reliably at that number and that presenting the plot invites a conclusion the data cannot support.
The respondent proposes that the plot be removed and replaced with a statement that publication bias could not be assessed.
The secretariat accepts this submission. Presenting an uninterpretable graphic is not a neutral act.
The funnel plot is removed for every outcome contributed by fewer than ten studies, and replaced by a statement that small-study effects could not be assessed at that number, together with the count of registered trials identified without posted results.
Studies using different outcome definitions are pooled into one estimate
Having read the draft synthesis on Does treatment with a glucagon-like peptide-1 receptor agonist increase residual gastric content at the time of elective procedural sedation?, the respondent puts one point to the committee.
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
This point is adjacent to the one made in submission 012 and the respondent puts it in a form the secretariat can act on.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.