Distorted Concept of "Accuracy", "Precision", and "Reliability"
Updated: Oct 10, 2022
Purpose:
To remind the Court of the concept of "reliability" used by the SCC in St-Onge.
To educate the Court as to the definitions of "accuracy", "precision", and "reliability" used by Brian Hodgson in his paper relied on by the SCC in St-Onge.
To focus on drift in accuracy and precision over time from:
the manufacturer's specifications,
and/or the accuracy and precision of the 8000 that was evaluated,
and/or this specific instrument when first placed into service,
as compared with accuracy and precision close to time of use.
Q. …But I want to suggest
to you that’s a different question from accuracy and precision
of the instrument and I think there’s the major point of
difference in your perspective from Mr. Kupferschmidt’s
perspective.
A. So, the acceptable range for a calibration
check is 90 to 110...
Q. Yes.
A. ...milligrams of alcohol in 100 millilitres
of blood so a result of 93 or 95 is considered to be within
that acceptable range, and when the instrument detects that
concentration during that cal-check, it allows testing to
proceed. If the cal-check is outside that acceptable range,
it will halt testing, and say ‘control outside tolerance.’
Q. So....
A. And the test will be terminated.
Q. So, you and the Centre determine accuracy
and precision based on policy as opposed to the definition
that Hodgson uses in the paper that was referred to in the
Supreme Court of Canada. You can go back and check that
definition if you like.
A. I might not have a copy of the Hodgson
paper.
Q. All right, I’m not sure if I have an extra
copy, but I can show you mine from Exhibit Number 2. It’s
Exhibit 2, Tab 2.
A. Ah, here we go. Okay. I’ve got a lot of
stuff here.
Q. Exhibit 2, Tab 2, page 85.
A. Eighty-five.
Q. Eighty-five has a definition of accuracy.
“Is the ability to measure a substance with a result that is
as close to the true quantity of that substance as possible.
For breath alcohol testing, this means measuring the breath
alcohol concentration as closely as possible to the true or
actual concentration of alcohol that is calculated to be in
the breath at the time of sampling.”
A. Correct.
Q. But if one wants to make a determination of
the accuracy of the instrument – I’m not talking about
accuracy of the procedure, but accuracy of the instrument, one
needs to take known samples, a number of them, using known –
known references, if I can use that, known alcohol standards,
run them through the instrument, a number of them, and take an
average. Compare that average with the value that one assumes
to be the true value. That’s how you determine accuracy of an
instrument. At least accuracy of the instrument for
instrument purposes, as opposed to accuracy of the subject
breath tests.
A. That sounds correct.
Q. Right. And with respect to precision,
“Precision is the ability to measure that same...” – and I’m
on page 86. “...is the ability to measure that same substance
repeatedly to produce the same result. For breath alcohol
testing, this means recorded measurement of the same
concentration of alcohol to produce results that are close to
each other as possible.” Well, that sounds a lot like
consistency.
A. Yes.
Q. But I want to suggest to you that the
determination – I mean, I think – I think I’m reading you as
saying the consistency from the Centre of Forensic Sciences’
perspective, the consistency is only with respect to the two
numbers of 93 and 95. It’s not with respect to consistency of
the instrument itself as determined by repeated – by a group
of control checks.
A. So, the accuracy or the ability of the
instrument to accurately record and measure the blood alcohol
concentration of a person who provides a sample is determined
by the calibration checks that are performed at the time of
testing. And so, those are deemed to be within the acceptable
range, then the accuracy of the breath tests based on that
standard are implied from that.
Q. I understand that that deeming....
A. And – sorry, and then there’s the two
samples that are taken in good agreement. All right? So,
that’s the second criteria is that the samples have to be
within 20 milligrams of alcohol in 100 millilitres of blood,
truncated. And they are.
Q. I understand your deeming.
A. Yes.
Q. I understand that policy, and it may in
fact, be a good policy for training of police officers, but
when it comes time in Court to access – assess accuracy and
precision I want to suggest you cannot assess instrumental
accuracy and instrumental precision based on just two values.
It’s impossible.
A. By that – yes, that’s correct. You would
need to do more analyses than that. And the accuracy and
precision of the instrument was determined at the time of the
evaluation by the Alcohol Test Committee.
Q. But not with this particular instrument.
A. Correct.
Q. And that was a long time ago.
A. That's correct. But, as the analogy I gave
before, is that two random instruments were tested back in
2007.
Q. Yes.
A. And the data from that applies to all the
instruments that come out from that, in the same way that the
alcohol standard solutions, when they come from the
manufacturer, are certified by us. We get a random lot.
Q. Yes.
A. And scientifically, it’s valid to take a
random sample of that, and that is applied – the data from
that is applied to the whole lot as an entity.
Q. They were random new instruments.
A. Yes.
Q. Recently calibrated by the manufacturer.
A. Yes.
[The cross-examiner should have clarified - But the issue is not accuracy, precision, and reliability at time of evaluation but rather:
1.this instrument
2. this instrument never so evaluated
3. long after 2007.]
Q. Right. Now, Hodgson says, at page 86,
third paragraph, “When evaluating instruments for accuracy and
precision, known concentrations of alcohol in water are used.
Accuracy refers to the ability of the instrument to measure
the breath alcohol concentration, to give readings that are
within plus or minus 5 percent of the true alcohol
concentration of known standard.” Now, I think there, Hodgson
is referring to the A-T-C evaluation procedures of plus or
minus 5 percent.
A. And it could be that that’s exactly what
he’s talking about with respect to all of these parameters,
accuracy, precision, reliability and specificity. That all
has to do with the evaluation parameters, not the operational
parameters.
Q. All right.
A. That’d be my interpretation.
Q. Then how is....
A. Without reading it entirely from the
beginning again, I can’t be more...
Q. Then how do....
A. ...conclusive than that.
Q. How does that help us? Using the Centre of
Forensic Sciences’ perspective on how you determine, deeming
accuracy and precision to be acceptable, how does that work
with this concept of reliability in Hodgson, which refers to
the ability of the instrument to perform over time, without
any significant drift in accuracy and precision. Because
obviously, the over time involves a comparison.
A. Correct.
Q. And you said that over time could mean
months, could mean years.
A. Yes. And the reliability of the instrument
is determined by the calibration check at the time of testing.
If there’s any gross drift over time, that would be detected
by the calibration check. Right?
Q. Even – even with respect to precision?
A. With respect to accuracy.
Q. What about precision? I mean, we have a
practical problem in Court now. We have to analyse the
question. I’m ultimately going to make submissions to the
Court to say there’s a problem with the reliability of this
instrument. So, reliability refers to the ability of the
instrument to perform over time without any significant drift
in accuracy and precision. Well, if we use as a reference
point the over time being at the time of the evaluation, we’ve
got lots of data on accuracy and precision of the evaluated
instruments.
A. Yes.
Q. Right? We have that from Terry Martin’s
paper.
A. Correct.
Q. We don’t have – I don’t know what data
we’ve got with respect to this Intoxilyzer 8000C when it was
first started being used, but we’ve got the practical problem
of what data do we use in Court to see if there has been or
has not been a drift in accuracy and precision, and what
you’re saying is, that there’s a policy, there’s a deeming....
A. A recommendation.
Q. A recommendation from the Centre of
Forensic Sciences that’s based wholly on just looking at two
pieces of data to make the determine [sic] of precision.
A. All right. Um, with respect to accuracy
and precision, the accuracy and precision is determined,
again, at the time of the instrument evaluation. The proper
operating of the instrument is determined at the time of
testing. With respect to the accuracy of that, it’s
determined from the calibration check. All right? So, the
acceptable variability associated with that calibration check
in the field is plus or minus 10 percent. We recognise that
there are external parameters that can cause the result not to
be 100 milligrams of alcohol in 100 millilitres of blood but
that could cause it to be 93 or 95. And so, given all that
variability we recognise that with that, that there’s going to
be some kind of change. If the calibration check is outside
the acceptable range, you’re not going to be able to proceed
with testing, and we know that the variability associated in
the field is always going to be greater – as you can see from
these numbers, if you were to take the last 10 values, and
then put those into a spreadsheet, you would see that the
accuracy would appear to be lower, but we don’t know that,
right? And the precision would be much greater because
there’s greater variability in these numbers compared to being
done in a laboratory and there always will be.
Comment:
It seems that the CFS "deem" accuracy and precision without drift over time simply as a result of the operational procedures at time of use. Where is the empirical proof for that hypothesis? It may be a technical norm or model that works for police, but is not good science. There is no empirical evidence that this deeming works. It is faith-based science. It does not foit the international literature.
#crossex #manufacturersspecifications #precision
#reliability #accuracy #precision #Hodgson #manufacturersspecifications