Draft scoping review: melanocortin receptor agonists — submissions
The 11 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
11 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The same concern is used to downgrade in two domains
The respondent notes that the review question — What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The draft downgrades for both indirectness and imprecision citing the same feature of the contributing trials. The respondent states that this double-counts a single concern and produces a rating lower than the framework supports.
The respondent proposes that the rating be raised by one level.
The secretariat accepts this submission in part. The double count is real in one of the two outcomes identified and not in the other, where the two domains rest on different features.
The rating for the first outcome has been raised by one level and the domain reasoning restated, and the reasoning for the second has been rewritten to make clear that the two downgrades rest on different features of the evidence.
The axis of the forest plot exaggerates a small difference
Having read the draft synthesis on What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are…, the respondent puts one point to the committee.
The plot is drawn on an axis spanning a narrow range, so an effect of no clinical consequence occupies most of the width of the figure. A reader who takes the visual impression rather than the number will overstate the finding.
The respondent proposes that a minimally important difference be marked on every forest plot where one has been established for the outcome.
The secretariat accepts the proposal where a minimally important difference exists and declines to invent one where it does not.
Forest plots now carry a marked minimally important difference wherever a published one exists for the outcome, with its source cited. Where none exists the figure says so beneath the axis.
The review does not say when it will be updated or what would trigger an update
The respondent’s comment on the draft review of What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… arises from comparing the included set against the respondent’s own knowledge of the field.
The respondent states that a synthesis in a field where trials report frequently is out of date from the day it is published, and that a reader has no way to know whether the version in front of them is current.
The respondent proposes a stated trigger for an update as well as a scheduled review date.
The secretariat accepts this submission. A review date alone tells a reader when the document will be looked at, not whether it should already have been.
Every synthesis now records a scheduled review date and a stated update trigger, being the reporting of a trial that would materially change the contributing evidence, and the registered trials capable of triggering an update are listed by name.
A funnel plot is presented for a set too small to interpret it
This submission concerns the draft review of What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The draft includes a funnel plot for an outcome contributed by fewer than ten studies. The respondent states that asymmetry cannot be assessed reliably at that number and that presenting the plot invites a conclusion the data cannot support.
The respondent proposes that the plot be removed and replaced with a statement that publication bias could not be assessed.
The secretariat accepts this submission. Presenting an uninterpretable graphic is not a neutral act.
The funnel plot is removed for every outcome contributed by fewer than ten studies, and replaced by a statement that small-study effects could not be assessed at that number, together with the count of registered trials identified without posted results.
The review question is drawn too broadly to be answerable
This is a submission on the draft review of What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are…, made from a statistical standpoint.
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
This submission is made in the same spirit as submission 002 and on a different aspect of the draft.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
The comparator was given at a dose below the one in general use
The respondent read the draft review of What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… and has confined this submission to a single matter.
Two contributing trials compare the intervention against an active comparator titrated to a dose lower than that reached in ordinary practice. Pooling them with trials using a full comparator dose produces an estimate that flatters the intervention.
The respondent proposes that comparator dose be tabulated in the included-studies table and that a sensitivity analysis restricted to full-dose comparators be reported.
The secretariat accepts this submission. Comparator dose is a condition of an effect estimate and was not being recorded.
Comparator dose is now a column in the included-studies table, and a sensitivity analysis restricted to comparators at the dose in general use is reported wherever the trials permit it.
Efficacy outcomes are rated for certainty and harms are not
The draft review of What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… was read against its registered protocol.
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
The document set should be published in translation
The respondent submits on the draft synthesis addressing What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
References should carry a persistent identifier for every cited source
The respondent has read the draft synthesis on What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… and makes one submission.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
Analytical surveys are treated as evidence about suppliers when they establish only what was in a sample
This submission concerns What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… and a convention used across the Institute’s output.
Several included surveys purchased material anonymously and analysed it. The respondent states that where the chain from a named supplier to the analysed vial cannot be documented, the result describes a sample and not a supplier.
The respondent proposes that provenance be an explicit eligibility dimension, with surveys reported separately according to whether it could be established.
Submission 008 concerns the same document. The respondent’s point is a different one.
The secretariat accepts this submission. Moderate certainty evidence from the included surveys supports statements about material circulating in a market; it does not support statements about any named supplier's output.
Provenance is now assessed for every included survey and reported as a study characteristic, surveys without establishable provenance are reported separately rather than pooled with those that have it, and no supplier-level inference is drawn from a sample-level result.
An indirect comparison is presented without an assessment of transitivity
This submission addresses the draft synthesis on What is the relationship between the indications in which melanocortin receptor agonists have been evaluated and the uses under which they are… from the standpoint of a reader who will use the summary and not the appendices.
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
Submission 009 concerns the same document. The respondent’s point is a different one.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.