Draft supplier evaluation rubric — submissions
The 10 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
10 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Two criteria can be satisfied without changing anything a supplier does
The respondent notes that supplier evaluation rubric produces a number that will be sorted, and submits with that in view.
The respondent identifies two criteria that are satisfied by publishing a document rather than by performing an activity, and states that both would be met by a supplier that changed only its website.
The respondent proposes that each criterion be tied to evidence of the underlying activity.
The secretariat accepts this submission. A criterion satisfied by a publication measures publication.
Both criteria are restated so that they are satisfied only by a document evidencing the underlying activity, such as a dated determination or a signed release record, and the rubric now states for every criterion what evidence satisfies it.
Declared interests should appear on the document rather than on a separate page
The respondent submits on supplier evaluation rubric. A supplier score will be read as a statement about product whatever the accompanying text says, and the design of the instrument has to take that as given.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The respondent notes submission 001 above and does not repeat the ground it covers.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
Two anchors on the same criterion can both describe the same supplier
The draft of supplier evaluation rubric was read by a respondent whose interest is in what a supplier could do to raise a score without changing anything else.
On the certificate criterion, a supplier reporting purity with a method but no content matches one anchor, and a supplier reporting purity with content but no method matches another. A supplier reporting neither matches both descriptions equally poorly.
The respondent proposes that anchors be rewritten as a checklist of present fields rather than as prose descriptions.
The secretariat accepts the diagnosis and adopts a narrower remedy.
The anchors are retained in prose, because a checklist would score a certificate carrying every field badly filled as though it were complete. Each anchor now names the fields that must be present for it to be reached, so that two anchors cannot describe one certificate.
Documentary practice is weighted equally with analytical evidence
This submission addresses supplier evaluation rubric from the standpoint of a reader who will see a score and not the rubric behind it.
The respondent states that a supplier can score well on documentation while releasing material on certificates generated by a third party it does not name, and that the two dimensions should not carry equal weight.
The respondent proposes that analytical evidence carry twice the weight of documentary practice.
The secretariat accepts this submission in part. The weighting is adjusted, though not to the ratio proposed, and a gate is added that the weighting alone would not achieve.
Analytical evidence now carries greater weight than documentary practice, and a supplier that does not identify the laboratory performing its determinations cannot reach the upper band regardless of its total score.
Whether a purchaser can obtain the record behind a figure is not scored
This is a submission on supplier evaluation rubric, made from an analytical standpoint.
A certificate reporting a purity figure is worth more where the chromatogram behind it can be obtained on request and less where it cannot. The rubric scores the certificate and not the availability of the record.
The respondent proposes that availability of the underlying record on written request be scored within the transparency criterion.
The secretariat accepts this submission.
Availability of the underlying record on written request is now an anchor within the transparency criterion, tested by request during each assessment cycle and recorded in the correspondence log.
Scores from different rubric versions are presented together
This submission concerns the draft of supplier evaluation rubric. The respondent works in a dispensing setting and sees how documents of this kind are used in practice.
The respondent states that dossiers assessed under an earlier version of the rubric appear in the same table as those assessed under the current one, and that the scores are not comparable.
The respondent proposes that the rubric version be carried with every score.
The respondent supports submission 005 so far as it goes and adds the matter set out here.
The secretariat accepts this submission. A score without its rubric version is not a measurement.
Every score now carries the rubric version and the assessment date, the comparison table displays only scores made under the current version, and dossiers not yet reassessed are shown as awaiting reassessment rather than with a stale score.
Point estimates are given without an interval
Having read the draft of supplier evaluation rubric, the respondent puts one point to the secretariat.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
An accreditation claim is scored without being verified
The respondent’s comment on supplier evaluation rubric arises from having assessed supplier documentation professionally.
The respondent states that the rubric awards credit for a claimed accreditation, and that accreditation claims in this sector are frequently unverifiable or refer to a scope that does not cover the determinations reported.
The respondent proposes that credit be awarded only where the accreditation and its scope are verified against the accrediting body's own register.
The secretariat accepts this submission. An unverified claim scored as though verified imports the claim into the Institute's own assessment.
Credit for accreditation is now awarded only where the Institute has verified the accreditation and its scope against the accrediting body's register, the verification date is recorded in the dossier, and an unverifiable claim is recorded as claimed and unverified with no credit.
Scores are reported to a precision the underlying judgements do not support
The respondent read supplier evaluation rubric in draft and has confined this submission to a single criterion.
The respondent states that scores are reported to one decimal place while the underlying criteria are judged on coarse ordinal scales, and that the precision is spurious.
The respondent proposes that scores be reported as whole numbers.
The secretariat accepts this submission in part. Reported precision is reduced. Scores are not rounded to whole numbers, because the resulting ties would themselves be read as findings of equivalence.
Scores are now reported to the precision the criteria support, the criterion-level judgements are published alongside the total so that the derivation can be checked, and the dossier states that a difference smaller than the reported precision is not a difference.
A high score invites a purchasing decision the rubric cannot support
The respondent has read supplier evaluation rubric in draft and makes one submission.
The rubric assesses documentation. A reader comparing two scores is making a purchasing decision, and the disclaimer accompanying the score does not prevent it because the disclaimer is read once and the score is read every time.
The respondent proposes that the scope of the score be printed in the same line as the score wherever it appears, rather than in a notice above the table.
The secretariat accepts this submission.
The scope of the score is now printed in the same row as the score on every surface that carries it, in the same weight, rather than in a separate notice.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.