Draft synthesis: Clinical evidence for growth-hormone… — submissions
The 9 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
9 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The document should not describe uses outside the approved indication
The respondent notes that the review question — What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research? — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
Intention-to-treat and efficacy-estimand results are combined without distinction
This is a submission on the draft review of What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research?, made from a statistical standpoint.
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
The comparator was given at a dose below the one in general use
The respondent submits on the draft synthesis addressing What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research?. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
Two contributing trials compare the intervention against an active comparator titrated to a dose lower than that reached in ordinary practice. Pooling them with trials using a full comparator dose produces an estimate that flatters the intervention.
The respondent proposes that comparator dose be tabulated in the included-studies table and that a sensitivity analysis restricted to full-dose comparators be reported.
The respondent has read submission 001 above and makes this submission independently of it.
The secretariat accepts this submission. Comparator dose is a condition of an effect estimate and was not being recorded.
Comparator dose is now a column in the included-studies table, and a sensitivity analysis restricted to comparators at the dose in general use is reported wherever the trials permit it.
A funnel plot is presented for a set too small to interpret it
The respondent has read the draft synthesis on What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research? and makes one submission.
The draft includes a funnel plot for an outcome contributed by fewer than ten studies. The respondent states that asymmetry cannot be assessed reliably at that number and that presenting the plot invites a conclusion the data cannot support.
The respondent proposes that the plot be removed and replaced with a statement that publication bias could not be assessed.
Submission 003 raises an adjacent matter. The respondent regards the two as separable and addresses only this one.
The secretariat accepts this submission. Presenting an uninterpretable graphic is not a neutral act.
The funnel plot is removed for every outcome contributed by fewer than ten studies, and replaced by a statement that small-study effects could not be assessed at that number, together with the count of registered trials identified without posted results.
The search date is not on the face of the document
The draft review of What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research? was read against its registered protocol.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
Results at materially different follow-up durations are pooled
This submission concerns the draft review of What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research?. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
The review does not say when it will be updated or what would trigger an update
The respondent read the draft review of What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research? and has confined this submission to a single matter.
The respondent states that a synthesis in a field where trials report frequently is out of date from the day it is published, and that a reader has no way to know whether the version in front of them is current.
The respondent proposes a stated trigger for an update as well as a scheduled review date.
The secretariat accepts this submission. A review date alone tells a reader when the document will be looked at, not whether it should already have been.
Every synthesis now records a scheduled review date and a stated update trigger, being the reporting of a trial that would materially change the contributing evidence, and the registered trials capable of triggering an update are listed by name.
Overlapping primary studies across included reviews are counted more than once
This submission addresses the draft synthesis on What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research? from the standpoint of a reader who will use the summary and not the appendices.
The draft is an overview of reviews and several included reviews share primary studies. The respondent states that the overview reports the total number of participants across reviews as though the sets were disjoint.
The respondent proposes that overlap be assessed and reported, and that no total be given without correcting for it.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 007.
The secretariat accepts this submission. An inflated participant total overstates the evidence base by an amount the reader cannot see.
Overlap between included reviews is now assessed at primary-study level and reported in a citation matrix, and participant totals are reported for the union of primary studies rather than as a sum across reviews.
Studies not published in English were excluded without assessment
The respondent’s comment on the draft review of What clinical evidence supports the uses under which growth-hormone secretagogues are supplied for research? arises from comparing the included set against the respondent’s own knowledge of the field.
The respondent states that a language restriction was applied at screening and that for some of the compounds in scope a substantial literature is published in other languages.
The respondent proposes that the restriction be removed and the review re-run.
The respondent has read submission 003 with interest and adds one observation the secretariat may find useful.
The secretariat accepts this submission in part. The restriction is removed prospectively and the records already excluded on language grounds have been retrieved and screened on title and abstract in translation. The full re-run proposed is not undertaken, and the limitation is recorded rather than concealed.
Language restriction is removed from the protocol for the series, the records previously excluded on that ground are listed with their screening outcome, and the residual limitation is stated in the abstract of this review.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.