Draft synthesis: Nationally registered neuropeptides and the… — submissions
The 16 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
16 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Two factual descriptions of the sponsor's programme are inaccurate
The respondent submits on For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable…, on a matter that is not specific to this draft but is visible in it.
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
A superseded version should remain reachable from the version that replaced it
The respondent read For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… in draft. The point applies to it and to the series generally.
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
Two included studies do not meet the registered eligibility criteria
This submission concerns the draft review of For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The respondent identifies two studies in the included set whose duration falls below the minimum stated in the protocol, and one excluded study that appears to meet every criterion.
The respondent proposes that the three be reassessed and that the outcome of the reassessment be recorded whichever way it goes.
The respondent has read submission 001 above and makes this submission independently of it.
The secretariat accepts this submission in part. On reassessment, one of the two included studies does fall below the minimum duration and has been removed. The second reports a duration that met the criterion in the protocol version registered at the time. The excluded study was excluded for a reason not clearly recorded, which was a documentation failure.
One study has been removed from the included set and the estimate recomputed, the exclusion reason for the third study has been corrected in the excluded-studies table, and the screening decisions are now recorded against the protocol version in force at the time of screening.
A surrogate outcome is used as the anchor without validation evidence
The respondent’s comment on the draft review of For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… arises from comparing the included set against the respondent’s own knowledge of the field.
The anchor outcome in the draft is a surrogate. The respondent states that the relationship between the surrogate and the outcome a decision turns on is itself an evidential question, and that the draft assumes it.
The respondent proposes that no surrogate serve as an anchor.
The secretariat accepts this submission in part. The anchor is retained where the surrogate is the only outcome the contributing trials measured, and the validation question is addressed rather than assumed.
Where the anchor is a surrogate, the synthesis now states the evidence for the surrogate relationship, rates it separately, and downgrades the anchor rating for indirectness accordingly rather than carrying the surrogate as though it were the outcome of interest.
Efficacy outcomes are rated for certainty and harms are not
Having read the draft synthesis on For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable…, the respondent puts one point to the committee.
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
The comparator was given at a dose below the one in general use
The respondent notes that the review question — For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
Two contributing trials compare the intervention against an active comparator titrated to a dose lower than that reached in ordinary practice. Pooling them with trials using a full comparator dose produces an estimate that flatters the intervention.
The respondent proposes that comparator dose be tabulated in the included-studies table and that a sensitivity analysis restricted to full-dose comparators be reported.
The secretariat accepts this submission. Comparator dose is a condition of an effect estimate and was not being recorded.
Comparator dose is now a column in the included-studies table, and a sensitivity analysis restricted to comparators at the dose in general use is reported wherever the trials permit it.
A sponsor trial meeting the eligibility criteria was excluded
This submission concerns For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… and makes one point.
The submission is made on behalf of the sponsor. It identifies a completed trial of the sponsor's compound that meets the stated eligibility criteria and does not appear in the included set, and supplies the trial report and the registry record.
The sponsor asks that the trial be included and the estimate recomputed. No view is expressed on the direction the recomputation should take.
The secretariat accepts this submission in part. The trial does meet the criteria and has been included. The recomputed estimate is materially unchanged, which the response states explicitly so that the outcome of the correction is on the record.
The trial is added to the included set, the estimate and the certainty rating have been recomputed, and the screening record now states why the trial was missed, which was a database indexing gap rather than a screening judgement. The submission is identified as an industry submission.
A sortable table implies a comparison the underlying data do not support
The draft review of For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… was read against its registered protocol.
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 002.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
Overlapping primary studies across included reviews are counted more than once
This submission addresses the draft synthesis on For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… from the standpoint of a reader who will use the summary and not the appendices.
The draft is an overview of reviews and several included reviews share primary studies. The respondent states that the overview reports the total number of participants across reviews as though the sets were disjoint.
The respondent proposes that overlap be assessed and reported, and that no total be given without correcting for it.
The respondent has read submission 003 with interest and adds one observation the secretariat may find useful.
The secretariat accepts this submission. An inflated participant total overstates the evidence base by an amount the reader cannot see.
Overlap between included reviews is now assessed at primary-study level and reported in a citation matrix, and participant totals are reported for the union of primary studies rather than as a sum across reviews.
The document should not describe uses outside the approved indication
The respondent has read the draft synthesis on For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… and makes one submission.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
Subgroup findings are reported that were not registered in the protocol
This submission concerns For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… and a convention used across the Institute’s output.
The draft reports differences between subgroups that do not appear in the registered protocol. The respondent states that unregistered subgroup analysis is hypothesis-generating and that the draft presents it in the same form as the pre-specified results.
The respondent proposes that unregistered analyses be removed.
The secretariat accepts this submission in part. The analyses are retained and relabelled rather than removed, because removing an analysis that was conducted leaves no record that it was.
Every subgroup analysis is now labelled as pre-specified or post hoc against the registered protocol, post hoc analyses are reported in a separate subsection without a certainty rating, and the protocol version against which the labelling was made is stated.
Absence of evidence is presented in a form a reader will take as negative evidence
The respondent submits on the draft synthesis addressing For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
The review protocol is described but its registration record is not linked
The respondent submits on For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable….
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
This submission should be read alongside submission 002, which arises on the same draft.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
Registered trials that never reported are not counted anywhere in the review
The respondent read the draft review of For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… and has confined this submission to a single matter.
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
The timepoint at which each outcome was extracted was chosen after the data were seen
The respondent has read For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable… in draft and makes a single submission.
The methods state that outcomes were extracted at the latest available timepoint. Several contributing trials report the same outcome at three timepoints, and taking the latest is a choice that can be made in the knowledge of what each shows.
The respondent proposes that the extraction timepoint be prespecified in the protocol, and that where it was not, the review report the estimate at every reported timepoint so that the choice is visible.
The secretariat accepts this submission. An extraction rule applied after the data are visible is a degree of freedom and should be recorded as one.
The extraction timepoint is now prespecified in the protocol for new reviews. Where a review predates the requirement, estimates are reported at every timepoint the contributing trials report.
The choice of effect measure is not justified and changes the appearance of the result
This is a submission on the draft review of For neuropeptides holding a national marketing registration outside the jurisdictions the Institute routinely reviews, what evidence is retrievable…, made from a statistical standpoint.
The draft reports relative effects for benefits and absolute effects for harms. The respondent states that the combination makes the finding look larger than the uniform presentation would, and that the choice should be justified or made uniform.
The respondent proposes that both relative and absolute effects be reported for every outcome.
The secretariat accepts this submission in part. Both measures are reported for every outcome where the baseline risk needed for the absolute effect can be stated. Where it cannot, the relative effect is reported alone with the reason.
Every outcome now reports the relative effect and, where an assumed baseline risk can be stated and sourced, the corresponding absolute effect, with the baseline risk and its source given in the same row.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.