Draft synthesis: Mitochondria-targeted peptides in… — submissions
The 12 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
12 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Crossover trials are pooled with parallel-group trials without adjustment
This is a submission on the draft review of What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline?, made from a statistical standpoint.
The analysis combines parallel-group and crossover designs by taking the first-period result from the crossovers, which discards half the data, or by taking the paired result, which is not comparable with an unpaired one. The methods do not say which was done.
The respondent proposes that the handling be stated and, where crossover data are used, that a sensitivity analysis excluding them be reported.
The secretariat accepts the requirement to state the handling and declines to require a sensitivity analysis in every case.
The methods section now states the handling of every non-parallel design. A sensitivity analysis is reported where crossover trials contribute more than a fifth of the weight.
Patient-reported outcomes are collected by the trials and not reported by the review
The respondent’s comment on the draft review of What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? arises from comparing the included set against the respondent’s own knowledge of the field.
The respondent states that several contributing trials measured quality of life and function and that the review reports neither, having selected outcomes on the basis of what could be pooled.
The respondent proposes that these outcomes be reported narratively where they cannot be pooled.
The respondent has read submission 001 with interest and adds one observation the secretariat may find useful.
The secretariat accepts this submission in part. The outcomes are reported narratively. The proposal to pool them is declined because the instruments used are not comparable and pooling would produce a figure with no interpretation.
Patient-reported outcomes measured by any contributing trial are now reported narratively by instrument, with the instrument named and its minimum important difference stated where one is published, and the absence of a pooled estimate explained.
The review protocol is described but its registration record is not linked
Having read the draft under consultation, which concerns What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline?, the respondent submits as follows.
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
The timepoint at which each outcome was extracted was chosen after the data were seen
The respondent read the draft review of What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? and has confined this submission to a single matter.
The methods state that outcomes were extracted at the latest available timepoint. Several contributing trials report the same outcome at three timepoints, and taking the latest is a choice that can be made in the knowledge of what each shows.
The respondent proposes that the extraction timepoint be prespecified in the protocol, and that where it was not, the review report the estimate at every reported timepoint so that the choice is visible.
The secretariat accepts this submission. An extraction rule applied after the data are visible is a degree of freedom and should be recorded as one.
The extraction timepoint is now prespecified in the protocol for new reviews. Where a review predates the requirement, estimates are reported at every timepoint the contributing trials report.
The review question is drawn too broadly to be answerable
The respondent has read What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? and submits on a matter of presentation.
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
The respondent notes that submission 004 has already been made and confines this submission to a matter not covered by it.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
Regulatory assessment documents were not searched
The draft review of What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? was read against its registered protocol.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
A continuity correction has been applied to zero-event arms without being reported
The respondent notes that the review question — What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? — is answerable only if the contributing trials measured the same thing, and submits with that in view.
Several contributing trials report no events in one arm. The pooled estimate implies that a correction was applied, since a ratio measure is otherwise undefined, but the methods do not name the correction or its magnitude, and the choice materially affects the result when events are rare.
The respondent proposes that the correction be named and that an alternative method be reported alongside for rare-event outcomes.
The secretariat accepts this submission. A correction that changes the estimate should not be invisible.
The correction is now named in the methods of every review in which one was applied, and rare-event outcomes carry a second estimate obtained by a method that does not require one.
Declared interests should appear on the document rather than on a separate page
This submission addresses the draft synthesis on What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? from the standpoint of a reader who will use the summary and not the appendices.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The respondent has read submission 007 and puts a further matter to the secretariat.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
Doses differing several-fold are pooled without examining dose-response
The respondent has read the draft synthesis on What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline? and makes one submission.
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
A single-trial result is presented in the visual form of a pooled estimate
Having read the draft synthesis on What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline?, the respondent puts one point to the committee.
An outcome contributed by one trial is presented in the same forest plot format as outcomes contributed by several. The respondent states that the format carries an implication of replication that a single trial does not have.
The respondent proposes that single-trial outcomes be presented differently and labelled as unreplicated.
The secretariat accepts this submission. The form of a graphic is part of what it asserts.
Outcomes contributed by a single trial are now reported without a pooled diamond, labelled as unreplicated, and downgraded for imprecision or inconsistency according to the framework rather than presented as a synthesis.
Risk-of-bias judgements are reported as an overall rating without the domains
The respondent submits on the draft synthesis addressing What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline?. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The respondent endorses the general approach taken in submission 003 and asks that it be extended to the matter identified here.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
The same concept is given three different names in one document
This submission concerns the draft review of What randomised evidence supports mitochondria-targeted peptides in primary mitochondrial disease and in age-related functional decline?. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The respondent notes that submission 002 has already been made and confines this submission to a matter not covered by it.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.