Stephen BissManipulating Control Test Data to Enhance Precision
Updated: Oct 5, 2022
Purpose:
To neutralize the government scientist's assertion that anomalous data should be excluded from calculation of precision.
To obtain an acknowledgment of drift in precision over time (see Hodgson definition of reliability)
To challenge the propriety of throwing out control test data outside 90 - 100 mg/100mls in any calculation of precision
This cross-examination shows the need for defence counsel to retain experts in statistics AND metrology.
This cross-examination shows the need for defence counsel to carefully review the study that the Crown expert is relying upon.
A. Just one thing before we get started. Let me
get organized here. On the previous day I spoke about a
paper. This was with respect to the data you referred to me
from South Simcoe Police Service.
Q. Yes.
A. Where the first calibration check was 88 and
then the subsequent ones were acceptable, and that it was a
phenomenon where we know that to happen with respect to
sometimes the first calibration check can be low, and that
wouldn't be improper to exclude that data point from the
calculation of the accuracy and precision of the instrument.
And I referred to a paper by Dubowski, and, in fact, it
turns out it's actually a paper by a gentleman of the name of
Hwang, H-W-A-N-G, et al, co-authors. And it's the Journal of
Analytical Toxicology, published in 2016, volume 40, page 338
to 344. And in that paper they refer to the fact that they
do a series of accuracy and precision testing that we talked
about, and that was shown in that case with respect to the
South Simcoe Police, and that they exclude the first point
because of they're aware of the limitations of the first
sample often being unacceptable due to
non-equaliberlation [sic] -- equilibration of the head space,
so they rely on nine data points, as opposed to ten.
Q. But, of course, they would never exclude
number two and number three and number four.
A. Correct.
Q. In other words, a significant proportion of
the data in any calculation and precision.
A. Yes.
Q. So if you're dealing with 50 points, it makes
no sense in doing a calculation of precision to exclude the
first 11, or 11 out of those 50?
A. Well, in this case... if I can find the data.
It wasn't the first 11 data points, but it was all the 11
data points that were below the low end of the acceptable
range of 90 milligrams of alcohol in 100 millilitres of
blood.
Q. Right. But on the last day you talked about
the phenomenon that during training you inform the officers,
the new qualified technicians or the recertifying --
requalified technicians, that sometimes you have a problem
with the first cal check being low because...
A. Yes.
Q. ...the system needs to warm up. Maybe they
haven't waited long enough for the whole system to -- is
"equilibrate" the right word?
A. Yes.
Q. To stabilize. To reach an equilibrium of
temperature so that Henry's law applies?
A. Correct.
Q. So that's with respect to one test, not eight,
nine, ten, eleven of them?
A. Well, what we were doing here with respect to
this data that you are referring to is different than what is
done during an accuracy and precision test with respect to
the Intoxilyzer 8000C.1. Here we took field data that had been
obtained at various times, various calibration checks, and so
as I explained earlier, that had any of these been part of an
actual breath test, breath testing wouldn't have been able to
proceed, and that the results are outside of the range that's
considered acceptable of 90 to 110 milligrams of alcohol in
100 millilitres of blood, and that's why they would be
excluded, because they're not representative of the range
that is accepted to be in.
Q. I see. But there's still control tests, and
we've got no indication, without looking at the accompanying
documentation, there was anything not functioning in the
simulator. There are control tests. We assume, without
digging further and actually seeing the officer's notes,
there are control tests that would appear to be with a proper
alcohol standard at the temperature shown in the data. The
temperature is there of 34 degrees plus or minus 0.2 degrees
Celsius. So why shouldn't there be at least an indication of
the precision of the instrument, to look at the last 50 cal
checks prior to my client's tests?
A. Because they are outside that acceptable
range. There's obviously some other issue that's going on
during those particular standalone calibration checks that we
don't know anything about, and because it's aberrant data,
you don't include it in the actual calculation.
Q. But the issue may be that the instrument is
producing consistently low calibration checks.
A. Well...
Q. And this is an indication that --
A. We know that -- sorry. Sorry. Go ahead.
Q. This is, this is an indication. That change
in precision is an indication to us. I mean, subject to
review of the documentation by the officer who was dealing
with it on that day, this is an indication of consistently
low cal checks.
A. And that's what the data that we went through
last time appears to show. But the instrument can't proceed
with testing. So in this case -- again, I don't remember how
many of these are associated with one particular case, but,
again, it speaks to a time other than when the tests in this
case were done. It speaks to another time in question that
you can't relate to the tests done in this case. And
normally, as I said, probably 99 percent of the time you get
a calibration check standalone and everything is fine,
diagnostics are fine and a self-test that's fine, and breath
testing proceeds to get two tests that are in good agreement
and game over. But occasionally there will be difficulties
associated with the breath tech trying to get the instrument
ready for the purposes of conducting breath tests. And as
we've talked about consistently, that sometimes that first
calibration check is below the acceptable range, in which
case something has to be done to try to get it back into the
acceptable range, whatever that might be. Sometimes it might
take more than one calibration check to get the instrument
into proper working order. It may take some additional
effort on the part of the qualified breath tech to get that
instrument ready and get the calibration check into the
acceptable range to proceed with testing, because if they
can't, then they're supposed to find another instrument.
Q. They're supposed to find another instrument
because...
A. Yes.
Q. ...it would then, at that point in time,
appear that the accuracy and precision of the instrument may
be jeopardized?
A. Well, there could be any number of
explanations why the data is low. That's one possible
reason, yes.
Q. And so that's why we should be looking at the
contemporaneous documentation [see footnote 2. below]. that went with those control
tests?
A. I would disagree with that. Again, that
material speaks to times other than when the test in question
here was done, and you can't relate that information to the
tests that were done in this case in any way -
mathematically, statistically or otherwise.
Q. Well, we've just talked about a method
statistically of relating it to the time of the subject test
in a calculation of precision. I mean, either those control
tests are relevant to a calculation of precision or they are
not, right?
A. That is correct. In this case those 11 points
are outside the acceptable range, and, therefore, would be
excluded from any kind of accuracy and precision calculation.
Q. But if we're trying to determine the why of
why it happened, especially when they aren't all that far off
from the low 90s that we're seeing in some of the other
calibration checks, the only way to make that determination,
to give a proper scientific opinion, is to look at the
contemporaneous documentation.2.
A. You need to look at that information for the
purposes of the breath test that was done in that particular
case. It's irrelevant to the test that was done in this
case.
Commentary:
The cross-examiner should have gone further and once again connected reliability with drift in accuracy and precision over time. An indication of drift in precision is therefore relevant to an assessment of reliability. A review of the contemporaneous documentation is therefore useful in studying whether or not there is a real drift in precision or rather some other cause for the anomalies.
1. Maybe the bigger problem is that no one is doing careful accuracy and precision tests (in an ISO 17025 certified lab) of approved instruments out in the field, including this particular instrument. Here we have at least an indication of a problem of precision. The instrument results are wandering. That problem requires disclosure of all accuracy and precision testing and contemporaneous documentation of anomalies.
2. See Motherisk Inquiry Report on Contemporaneous Documentation and ISO 17025.