Draft synthesis: Kidney outcomes with incretin receptor… — submissions
The 11 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
11 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Crossover trials are pooled with parallel-group trials without adjustment
The respondent read the draft review of In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes? and has confined this submission to a single matter.
The analysis combines parallel-group and crossover designs by taking the first-period result from the crossovers, which discards half the data, or by taking the paired result, which is not comparable with an unpaired one. The methods do not say which was done.
The respondent proposes that the handling be stated and, where crossover data are used, that a sensitivity analysis excluding them be reported.
The secretariat accepts the requirement to state the handling and declines to require a sensitivity analysis in every case.
The methods section now states the handling of every non-parallel design. A sensitivity analysis is reported where crossover trials contribute more than a fifth of the weight.
The document is unreadable without specialist training
The respondent’s comment on the draft review of In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes? arises from comparing the included set against the respondent’s own knowledge of the field.
The respondent states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
The choice of effect measure is not justified and changes the appearance of the result
This submission concerns the draft review of In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes?. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The draft reports relative effects for benefits and absolute effects for harms. The respondent states that the combination makes the finding look larger than the uniform presentation would, and that the choice should be justified or made uniform.
The respondent proposes that both relative and absolute effects be reported for every outcome.
The secretariat accepts this submission in part. Both measures are reported for every outcome where the baseline risk needed for the absolute effect can be stated. Where it cannot, the relative effect is reported alone with the reason.
Every outcome now reports the relative effect and, where an assumed baseline risk can be stated and sourced, the corresponding absolute effect, with the baseline risk and its source given in the same row.
The review protocol is described but its registration record is not linked
The draft review of In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes? was read against its registered protocol.
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
The search was limited by publication date and the limit is not justified
The respondent notes that the review question — In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes? — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The search strategy carries a start date some years before the review. No reason is given. If the limit reflects the date of first synthesis of the compound it is defensible and should be said; if it reflects convenience it is a source of bias.
The respondent proposes that every date limit carry its justification in the search-strategy section.
The secretariat accepts the requirement to justify and declines to remove limits that are justified.
Every date limit in the series now carries its justification in the search-strategy section. Limits that could not be justified on re-examination have been removed and the affected searches re-run.
Intention-to-treat and efficacy-estimand results are combined without distinction
Having read the draft synthesis on In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes?, the respondent puts one point to the committee.
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
This submission is made in the same spirit as submission 002 and on a different aspect of the draft.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
Regulatory assessment documents were not searched
The respondent submits on the draft synthesis addressing In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes?. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
Statistical heterogeneity is treated as though it measured clinical heterogeneity
The respondent has read the draft synthesis on In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes? and makes one submission.
The draft reports a heterogeneity statistic and proceeds to pool where it is low. The respondent states that a low statistic in a small set of trials is uninformative, and that clinical and methodological similarity should be assessed before any statistic is consulted.
The respondent proposes that the decision to pool be justified on clinical grounds first and that the statistic be reported as a description rather than used as a threshold.
The secretariat accepts this submission in part. The decision to pool is now made on clinical and methodological grounds and stated as such. The statistic continues to be reported, because readers expect it and its absence would be read as concealment.
The decision to pool is now justified in prose against the population, intervention, comparator and outcome before any statistic is presented, and the heterogeneity statistic is reported with its confidence interval and a note of the number of contributing studies.
A sortable table implies a comparison the underlying data do not support
This submission addresses the draft synthesis on In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes? from the standpoint of a reader who will use the summary and not the appendices.
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
Submission 005 concerns the same document. The respondent’s point is a different one.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
Doses differing several-fold are pooled without examining dose-response
This is a submission on In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes?.
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
Risk-of-bias judgements are reported as an overall rating without the domains
This is a submission on the draft review of In adults with type 2 diabetes and chronic kidney disease, what is the effect of a glucagon-like peptide-1 receptor agonist on kidney outcomes?, made from a statistical standpoint.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.